Insights from Web Directions Summit 2023

17 Nov 2023 Insights from Web Directions Summit 2023

This post was co-written by Shine software engineers Sampurna Sen, Loki Martin and Raimee Abbassi

A couple of weeks ago we had the opportunity to go to Web Directions Summit 2023 in Sydney, Australia. It was an event that had us all feeling charged-up like programmers in a caffeine shop!

The main theme of this year’s conference was (of course) AI, that mysterious thing that might one day make our toasters smarter than we are. Talks covered topics ranging from generative AI and Web Machine Learning through to more traditional areas like accessibility, design systems and all things CSS and JS.

What did we learn? Well, for starters future tech is probably going to be a lot smarter than we think! Beyond that, a single blog post won’t be able to cover all of the topics we heard about, so we have instead chosen to focus here on what we learnt in the areas of AI and accessibility.

Expanding Dark Forest and Generative AI 

The Dark Forest Theory of the Internet posits that due to the prominence of antagonistic behaviour on the internet, people are more inclined to move away from public social media to more private groups, encrypted messaging apps and other less visible means of communication. Coupled with the rapid growth of generative AI, the dark forest (or the public web space) is poised to become much, much darker.  

Generative agents and AI-driven software entities are designed to simulate nuanced human behaviours and interactions, and this has evolved rapidly – at a much faster rate than expected –  for large language models (LLMs) and image generators. Consequently, we may soon see the internet flooded with AI-generated content, blurring the lines between human and machine. 

So what does this new internet look like? We’re already familiar with the plethora of bots and fake content that occupy substantial corners of the online world, but we’’re on the brink of this escalating exponentially. Essentially, consider the polarisation, misinformation and disinformation we see online today and multiply it by 10. As keynote speaker Maggie Appleton put it: “we are about to drown in a sea of informational garbage”.

The Mediated Web

If this all sounds a bit grim, note that the general sentiment of the summit was cautiously optimistic. There were a lot of ideas thrown around about how we can combat the downsides of generative AI. For example, the concept of a “mediated web” holds the potential to shape the future of the internet. This emerging paradigm envisions a future where user agents, essentially software entities, act as proxies for users, executing actions on their behalf. 

This concept aligns with the fundamental principle of cybernetics, specifically the second axiom which states that systems engaging in information exchange naturally couple (structural coupling) into a greater unity. In the context of humanity, this ongoing communication and mutual learning already exists, with the output of one serving as the input for another. We call it socialisation. It just feels like it’s the first time because of its scale and depth. 

We are now entering into a new range of deep informational relationships with learning systems that aren’t just human. So we’re learning about learning and we are embedding that learning about learning into systems creating a feedback loop where our evolving understanding becomes the foundation for future inputs, enhancing the collective learning experience. This cycle resembles an Ouroboros, a serpent eating its own tail to continually nourish and sustain itself:

When machine learning assists our own learning

With this cycle of constant evolution and learning , we have started seeing a lot of companies embrace machine learning and create their own offerings. We can use these tools to accelerate our own learning, so the sooner we become familiar with them the better. The most relevant for software professionals are:

Creative tools

These are some generative AI tools to create images from natural language descriptions or prompts. They are commonly used for presentations and creating professional photos.

• Midjourney AI
• DALL-E
• Canva text to image generator

Conversational Chatbots

These are great for ideation and generating boilerplate code. They are also useful for debugging and code explanations.

• ChatGPT
• Bing Chat
• Google Bard
• V0 – generates copy-and-paste friendly HTML or React code

Extensions that integrate with your IDE 

These are supercharged autocomplete extensions that integrate to an IDE (Visual Studio code for example). Their contextual awareness makes these tools highly beneficial for coding. It feels like engaging in pair programming with a skilled coding partner who can provide valuable assistance.

• Github Copilot – Natural language text and source code from public repositories, such as GitHub 
• Code Llama – Trained on code-specific datasets and code-related data. 
• Amazon CodeWhisperer
• Tab Nine
• Ask Codi

WebML – Machine Learning in the Browser

Most of us assume machine learning is a backend focused task. But machine learning in the browser is becoming a reality. W3C Launched the Web Machine Learning Working Group in April 2021 and these tools and libraries are making AI more accessible and efficient on the web. Some common libraries out there are:

• Neural Network for Web – pose detection 
• CoCo-ssd model – object detection
• Replicate – npm package to create your own AI applications
• Some open source tools that offer powerful features and capabilities for machine learning and deep learning:

• ONNX – better choice for projects that require portability and compatibility with other frameworks
• TensorFlow – better for projects that require more flexibility and customization
• LangChain JS – framework for developing applications powered by language models

There are several advantages of implementing machine learning in web browsers. These benefits include reduced latency, cost savings for server resources, enhanced privacy, and the utilisation of Node.js as a just-in-time compiler.

Multi-modal Architectures for AI Agents

This is a design approach where an AI system utilises a combination of multiple models, each specialised in different tasks or domains. Instead of relying on a single monolithic model, multi-model architectures leverage the strengths of individual models for specific tasks, allowing the AI agent to switch between them as needed.

Fuyu-8B is a multi-modal text and image transformer trained by Adept AI. More information available here https://www.adept.ai/blog/fuyu-8b.

The downsides of software engineers using AI tools

As a software engineer, when considering whether to recommend using AI, it’s essential to weigh the pros and cons. While AI has made significant advancements and offers some benefits, there are some notable drawbacks that should be taken into account.

One of the cons of using AI in software engineering is its potential for inaccuracies. AI systems are not infallible and can sometimes produce incorrect results or predictions, which can lead to problems if not carefully monitored.

Another limitation is its limited context. AI models may not fully understand the broader context of a problem, and this can result in suboptimal decision-making or problem-solving in certain situations.

Furthermore, the use of AI carries the risk of leaking sensitive information, which can have serious consequences for both individuals and organisations. Additionally, there are security risks associated with AI. AI systems can become targets for malicious actors who aim to exploit vulnerabilities in the software.

At the conference we learnt about security vulnerabilities, such as encoded XSS (Cross-Site Scripting) inside images, and tactics used by hackers to hide malicious scripts. ChatGPT once generated a response about an npm package that does not exist. A hacker immediately took advantage and created one for that package that included a malicious script. This emphasises the importance of staying vigilant and proactive in web security when using AI.

Given these challenges, a better way to consider using AI as a software engineer may be to implement a “chain of thought prompting” approach. Use AI to assist, prompt, or augment human decision-making and problem-solving rather than relying solely on AI for critical tasks. By combining the strengths of AI with human expertise, we can leverage AI as a powerful tool while mitigating some of its inherent limitations and risks.

Being an accessibility ally with generative AI

In software development, “accessibility” is often abbreviated as “a11y”, a numeronym where 11 represents the count of letters between the letter a and the letter y:

“a11y” can also be read as “ally”, which we think is an appropriate description for somebody who keeps accessibility at the heart of their design and development considerations. 

At Web Directions Summit there was a big focus on the topic of accessibility. This makes sense, because 20% of Australians identify as a person with a disability. This is 1 out of 5 people who need information that may be available online and who rely on devices to do things like order food, buy stuff, pay taxes, apply for jobs, and talk with friends and family. Technology built with accessibility in mind has made things easier for people with disabilities. 

With the use of generative AI, we know that technology will evolve much faster, but will it still be useful for the 20%? We need to step back for a bit and ask ourselves, are we an “ally”? Do we develop applications with accessibility in mind?  Is accessibility baked into our design systems? How can we prevent a UX gap for people with disabilities when digital teams, armed with powerful AI tools, develop and deliver these amazing solutions at a much faster rate?  Simon Mateljan, a Design Manager at Atlassian, gave a great talk on how to bake accessibility into design systems. Specifically: 

1. Team Structure – The team should have an empowered champion or champions for accessibility. 
2. Education – Your team should close the gap on the knowledge about accessibility and what is required for your customers. This can be done through training, and familiarity with the WCAG guidelines. 
3. Audit – Understand your application or system’s current state then set a baseline. Decide the scale of the audit. After the audit is done, your team can prioritise and work on improvements. 
4. Foundations – Foundations should be set for various aspects of your team’s or company’s design system – for example: foundations on colour, spacing, coding, documentation, and experience. It should be flexible, scalable and maintainable. 

Making our apps and systems accessible translates to inclusion and better opportunities. It means more people can use them. For the dollar bottom line, recent research has revealed that there are billions of dollars in potential revenue if accessibility needs are accommodated. We can use various AI solutions like conversational AI programs which can be incorporated into our app or browser to assist people with screen reading and voice prompts. Designers can have a design “co-pilot” that will give accessibility advice when they are designing UI components. There’s plenty of solutions that we can think about, but first we need to bake accessibility into our thought process, design and implementation. We can greatly impact other people’s lives in a positive way –  the forest becomes less dark as we shed light within. 🙂

Wrapping Up

Apart from all the nerdy stuff, we also enjoyed some lighthearted moments from the summit, including a few humorous pick-up lines for those in the web development game:

• Are you https? Because without you I am insecure ://
• Are you React? Because you make me render.
• Are you a lifecycle hook? Because my component did mount. (That’s another one for React developers)

In conclusion, Web Directions Summit 2023 provided valuable insights into the future of the web, emerging technologies, and best practices in web development. It emphasised the importance of staying informed and adaptable in the ever-evolving web landscape. The imminent expansion of the “dark forest”, although scary, is also a chance for us as software engineers to lead the way in shaping the digital wilderness into a landscape where technology enhances the human experience rather than diminishing it.

In short, we laughed, we learned, and we left determined to stay curious, stay innovative, and most importantly, stay connected in the AI-driven web universe!