Simon Willison's Weblog: accessibility

Introducing ChatGPT Atlas

2025-10-21T18:45:13+00:00

Last year OpenAI hired Chrome engineer Darin Fisher, which sparked speculation they might have their own browser in the pipeline. Today it arrived.

ChatGPT Atlas is a Mac-only web browser with a variety of ChatGPT-enabled features. You can bring up a chat panel next to a web page, which will automatically be populated with the context of that page.

The "browser memories" feature is particularly notable, described here:

If you turn on browser memories, ChatGPT will remember key details from your web browsing to improve chat responses and offer smarter suggestions—like retrieving a webpage you read a while ago. Browser memories are private to your account and under your control. You can view them all in settings, archive ones that are no longer relevant, and clear your browsing history to delete them.

Atlas also has an experimental "agent mode" where ChatGPT can take over navigating and interacting with the page for you, accompanied by a weird sparkle overlay effect:

Here's how the help page describes that mode:

In agent mode, ChatGPT can complete end to end tasks for you like researching a meal plan, making a list of ingredients, and adding the groceries to a shopping cart ready for delivery. You're always in control: ChatGPT is trained to ask before taking many important actions, and you can pause, interrupt, or take over the browser at any time.

Agent mode runs also operates under boundaries:

System access: Cannot run code in the browser, download files, or install extensions.

Data access: Cannot access other apps on your computer or your file system, read or write ChatGPT memories, access saved passwords, or use autofill data.

Browsing activity: Pages ChatGPT visits in agent mode are not added to your browsing history.

You can also choose to run agent in logged out mode, and ChatGPT won't use any pre-existing cookies and won't be logged into any of your online accounts without your specific approval.

These efforts don't eliminate every risk; users should still use caution and monitor ChatGPT activities when using agent mode.

I continue to find this entire category of browser agents deeply confusing.

The security and privacy risks involved here still feel insurmountably high to me - I certainly won't be trusting any of these products until a bunch of security researchers have given them a very thorough beating.

I'd like to see a deep explanation of the steps Atlas takes to avoid prompt injection attacks. Right now it looks like the main defense is expecting the user to carefully watch what agent mode is doing at all times!

Update: OpenAI's CISO Dane Stuckey provided exactly that the day after the launch.

I also find these products pretty unexciting to use. I tried out agent mode and it was like watching a first-time computer user painstakingly learn to use a mouse for the first time. I have yet to find my own use-cases for when this kind of interaction feels useful to me, though I'm not ruling that out.

There was one other detail in the announcement post that caught my eye:

Website owners can also add ARIA tags to improve how ChatGPT agent works for their websites in Atlas.

Which links to this:

ChatGPT Atlas uses ARIA tags---the same labels and roles that support screen readers---to interpret page structure and interactive elements. To improve compatibility, follow WAI-ARIA best practices by adding descriptive roles, labels, and states to interactive elements like buttons, menus, and forms. This helps ChatGPT recognize what each element does and interact with your site more accurately.

A neat reminder that AI "agents" share many of the characteristics of assistive technologies, and benefit from the same affordances.

The Atlas user-agent is Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 - identical to the user-agent I get for the latest Google Chrome on macOS.

Via Hacker News

Tags: accessibility, aria, browsers, chrome, privacy, security, ai, openai, prompt-injection, generative-ai, ai-agents, browser-agents

Should form labels be wrapped or separate?

2025-10-17T18:25:45+00:00

Should form labels be wrapped or separate?

James Edwards notes that wrapping a form input in a label event like this has a significant downside:

<label>Name <input type="text"></label>

It turns out both Dragon Naturally Speaking for Windows and Voice Control for macOS and iOS fail to understand this relationship!

You need to use the explicit <label for="element_id"> syntax to ensure those screen readers correctly understand the relationship between label and form field. You can still nest the input inside the label if you like:

<label for="idField">Name
  <input id="idField" type="text">
</label>

Via Chris Ferdinandi

Tags: accessibility, html, screen-readers

New dashboard: alt text for all my images

2025-04-28T01:22:27+00:00

New dashboard: alt text for all my images

I got curious today about how I'd been using alt text for images on my blog, and realized that since I have Django SQL Dashboard running on this site and PostgreSQL is capable of parsing HTML with regular expressions I could probably find out using a SQL query.

I pasted my PostgreSQL schema into Claude and gave it a pretty long prompt:

Give this PostgreSQL schema I want a query that returns all of my images and their alt text. Images are sometimes stored as HTML image tags and other times stored in markdown.

blog_quotation.quotation, blog_note.body both contain markdown. blog_blogmark.commentary has markdown if use_markdown is true or HTML otherwise. blog_entry.body is always HTML

Write me a SQL query to extract all of my images and their alt tags using regular expressions. In HTML documents it should look for either <img .* src="..." .* alt="..." or <img alt="..." .* src="..." (images may be self-closing XHTML style in some places). In Markdown they will always be ![alt text](url)

I want the resulting table to have three columns: URL, alt_text, src - the URL column needs to be constructed as e.g. /2025/Feb/2/slug for a record where created is on 2nd feb 2025 and the slug column contains slug

Use CTEs and unions where appropriate

It almost got it right on the first go, and with a couple of follow-up prompts I had the query I wanted. I also added the option to search my alt text / image URLs, which has already helped me hunt down and fix a few old images on expired domain names. Here's a copy of the finished 100 line SQL query.

Tags: accessibility, alt-text, postgresql, sql, ai, django-sql-dashboard, generative-ai, llms, ai-assisted-programming, claude

Notes from my Accessibility and Gen AI podcast appearance

2025-03-02T14:51:43+00:00

I was a guest on the most recent episode of the Accessibility + Gen AI Podcast, hosted by Eamon McErlean and Joe Devon. We had a really fun, wide-ranging conversation about a host of different topics. I've extracted a few choice quotes from the transcript.

LLMs for drafting alt text

I use LLMs for the first draft of my alt text (22:10):

I actually use Large Language Models for most of my alt text these days. Whenever I tweet an image or whatever, I've got a Claude project called Alt text writer. It's got a prompt and an example. I dump an image in and it gives me the alt text.

I very rarely just use it because that's rude, right? You should never dump text onto people that you haven't reviewed yourself. But it's always a good starting point.

Normally I'll edit a tiny little bit. I'll delete an unimportant detail or I'll bulk something up. And then I've got alt text that works.

Often it's actually got really good taste. A great example is if you've got a screenshot of an interface, there's a lot of words in that screenshot and most of them don't matter.

The message you're trying to give in the alt text is that it's two panels on the left, there's a conversation on the right, there's a preview of the SVG file or something. My alt text writer normally gets that right.

It's even good at summarizing tables of data where it will notice that actually what really matters is that Gemini got a score of 57 and Nova got a score of 53 - so it will pull those details out and ignore [irrelevant columns] like the release dates and so forth.

Here's the current custom instructions prompt I'm using for that Claude Project:

You write alt text for any image pasted in by the user. Alt text is always presented in a fenced code block to make it easy to copy and paste out. It is always presented on a single line so it can be used easily in Markdown images. All text on the image (for screenshots etc) must be exactly included. A short note describing the nature of the image itself should go first.

Is it ethical to build unreliable accessibility tools?

On the ethics of building accessibility tools on top of inherently unreliable technology (5:33):

Some people I've talked to have been skeptical about the accessibility benefits because their argument is that if you give somebody unreliable technology that might hallucinate and make things up, surely that's harming them.

I don't think that's true. I feel like people who use screen readers are used to unreliable technology.

You know, if you use a guide dog - it's a wonderful thing and a very unreliable piece of technology.

When you consider that people with accessibility needs have agency, they can understand the limitations of the technology they're using. I feel like giving them a tool where they can point their phone at something and it can describe it to them is a world away from accessibility technology just three or four years ago.

Why I don't feel threatened as a software engineer

This is probably my most coherent explanation yet of why I don't see generative AI as a threat to my career as a software engineer (33:49):

My perspective on this as a developer who's been using these systems on a daily basis for a couple of years now is that I find that they enhance my value. I am so much more competent and capable as a developer because I've got these tools assisting me. I can write code in dozens of new programming languages that I never learned before.

But I still get to benefit from my 20 years of experience.

Take somebody off the street who's never written any code before and ask them to build an iPhone app with ChatGPT. They are going to run into so many pitfalls, because programming isn't just about can you write code - it's about thinking through the problems, understanding what's possible and what's not, understanding how to QA, what good code is, having good taste.

There's so much depth to what we do as software engineers.

I've said before that generative AI probably gives me like two to five times productivity boost on the part of my job that involves typing code into a laptop. But that's only 10 percent of what I do. As a software engineer, most of my time isn't actually spent with the typing of the code. It's all of those other activities.

The AI systems help with those other activities, too. They can help me think through architectural decisions and research library options and so on. But I still have to have that agency to understand what I'm doing.

So as a software engineer, I don't feel threatened. My most optimistic view of this is that the cost of developing software goes down because an engineer like myself can be more ambitious, can take on more things. As a result, demand for software goes up - because if you're a company that previously would never have dreamed of building a custom CRM for your industry because it would have taken 20 engineers a year before you got any results... If it now takes four engineers three months to get results, maybe you're in the market for software engineers now that you weren't before.

Tags: accessibility, alt-text, podcasts, ai, generative-ai, llms, podcast-appearances

Why are my live regions not working?

2025-01-08T03:54:21+00:00

Why are my live regions not working?

Useful article to help understand ARIA live regions. Short version: you can add a live region to your page like this:

<div id="notification" aria-live="assertive"></div>

Then any time you use JavaScript to modify the text content in that element it will be announced straight away by any screen readers - that's the "assertive" part. Using "polite" instead will cause the notification to be queued up for when the user is idle instead.

There are quite a few catches. Most notably, the contents of an aria-live region will usually NOT be spoken out loud when the page first loads, or when that element is added to the DOM. You need to ensure the element is available and not hidden before updating it for the effect to work reliably across different screen readers.

I got Claude Artifacts to help me build a demo for this, which is now available at tools.simonwillison.net/aria-live-regions. The demo includes instructions for turning VoiceOver on and off on both iOS and macOS to help try that out.

Via Comment on Hacker News

Tags: accessibility, aria, javascript, screen-readers, ai-assisted-programming, claude-artifacts, prompt-to-app

Clay UI library

2024-12-21T23:12:17+00:00

Clay UI library

Fascinating project by Nic Barker, who describes Clay like this:

Clay is a flex-box style UI auto layout library in C, with declarative syntax and microsecond performance.

His intro video to the library is outstanding: I learned a ton about how UI layout works from this, and the animated visual explanations are clear, tasteful and really helped land the different concepts:

Clay is a C library delivered in a single ~2000 line clay.h dependency-free header file. It only handles layout calculations: if you want to render the result you need to add an additional rendering layer.

In a fascinating demo of the library, the Clay site itself is rendered using Clay C compiled to WebAssembly! You can even switch between the default HTML renderer and an alternative based on Canvas.

This isn't necessarily a great idea: because the layout is entirely handled using <div> elements positioned using transform: translate(0px, 70px) style CSS attempting to select text across multiple boxes behaves strangely, and it's not clear to me what the accessibility implications are.

Update: Matt Campbell:

The accessibility implications are as serious as you might guess. The links aren't properly labeled, there's no semantic markup such as headings, and since there's a div for every line, continuous reading with a screen reader is choppy, that is, it pauses at the end of every physical line.

It does make for a very compelling demo of what Clay is capable of though, especially when you resize your browser window and the page layout is recalculated in real-time via the Clay WebAssembly bridge.

You can hit "D" on the website and open up a custom Clay debugger showing the hierarchy of layout elements on the page:

This also means that the entire page is defined using C code! Given that, I find the code itself surprisingly readable

void DeclarativeSyntaxPageDesktop() {
  CLAY(CLAY_ID("SyntaxPageDesktop"), CLAY_LAYOUT({ .sizing = { CLAY_SIZING_GROW(), CLAY_SIZING_FIT({ .min = windowHeight - 50 }) }, .childAlignment = {0, CLAY_ALIGN_Y_CENTER}, .padding = {.x = 50} })) {
    CLAY(CLAY_ID("SyntaxPage"), CLAY_LAYOUT({ .sizing = { CLAY_SIZING_GROW(), CLAY_SIZING_GROW() }, .childAlignment = { 0, CLAY_ALIGN_Y_CENTER }, .padding = { 32, 32 }, .childGap = 32 }), CLAY_BORDER({ .left = { 2, COLOR_RED }, .right = { 2, COLOR_RED } })) {
      CLAY(CLAY_ID("SyntaxPageLeftText"), CLAY_LAYOUT({ .sizing = { CLAY_SIZING_PERCENT(0.5) }, .layoutDirection = CLAY_TOP_TO_BOTTOM, .childGap = 8 })) {
        CLAY_TEXT(CLAY_STRING("Declarative Syntax"), CLAY_TEXT_CONFIG({ .fontSize = 52, .fontId = FONT_ID_TITLE_56, .textColor = COLOR_RED }));
        CLAY(CLAY_ID("SyntaxSpacer"), CLAY_LAYOUT({ .sizing = { CLAY_SIZING_GROW({ .max = 16 }) } })) {}
        CLAY_TEXT(CLAY_STRING("Flexible and readable declarative syntax with nested UI element hierarchies."), CLAY_TEXT_CONFIG({ .fontSize = 28, .fontId = FONT_ID_BODY_36, .textColor = COLOR_RED }));
        CLAY_TEXT(CLAY_STRING("Mix elements with standard C code like loops, conditionals and functions."), CLAY_TEXT_CONFIG({ .fontSize = 28, .fontId = FONT_ID_BODY_36, .textColor = COLOR_RED }));
        CLAY_TEXT(CLAY_STRING("Create your own library of re-usable components from UI primitives like text, images and rectangles."), CLAY_TEXT_CONFIG({ .fontSize = 28, .fontId = FONT_ID_BODY_36, .textColor = COLOR_RED }));
      }
      CLAY(CLAY_ID("SyntaxPageRightImage"), CLAY_LAYOUT({ .sizing = { CLAY_SIZING_PERCENT(0.50) }, .childAlignment = {.x = CLAY_ALIGN_X_CENTER} })) {
        CLAY(CLAY_ID("SyntaxPageRightImageInner"), CLAY_LAYOUT({ .sizing = { CLAY_SIZING_GROW({ .max = 568 }) } }), CLAY_IMAGE({ .sourceDimensions = {1136, 1194}, .sourceURL = CLAY_STRING("/clay/images/declarative.png") })) {}
      }
    }
  }
}

I'm not ready to ditch HTML and CSS for writing my web pages in C compiled to WebAssembly just yet, but as an exercise in understanding layout engines (and a potential tool for building non-web interfaces in the future) this is a really interesting project to dig into.

To clarify here: I don't think the web layout / WebAssembly thing is the key idea behind Clay at all - I think it's a neat demo of the library, but it's not what Clay is for. It's certainly an interesting way to provide a demo of a layout library!

Nic confirms:

You totally nailed it, the fact that you can compile to wasm and run in HTML stemmed entirely from a “wouldn’t it be cool if…” It was designed for my C projects first and foremost!

Via Hacker News

Tags: accessibility, c, css, html, webassembly

Themes from DjangoCon US 2024

2024-09-27T23:36:02+00:00

I just arrived home from a trip to Durham, North Carolina for DjangoCon US 2024. I’ve already written about my talk where I announced a new plugin system for Django; here are my notes on some of the other themes that resonated with me during the conference.

Growing the Django Software Foundation (DSF)

Jacob Kaplan-Moss gave my favorite talk of the conference, asking what the Django Software Foundation could do if it quadrupled its annual income from $250,000 to $1 million dollars, and then mapping out a convincing path to get there.

I really liked this diagram Jacob provided summarizing the foundation’s current income and expenditures. It’s pretty cool that $90,000 of annual income comes from individual donors, over a third of the total since corporate donors provide $160,000.

Top priority would be hiring an Executive Director for the foundation, which is currently lead entirely by an elected, volunteer board. I’ve seen how useful a professional ED is from my own experiences on the Python Software Foundation board.

Having someone working full time on the foundation outside of our current fellows - who have more than enough on their plates already - would enable the foundation to both take on more ambitious goals and also raise more money with which to tackle them.

A line that Jacob used repeatedly in his talk about funding the foundation was this: if you or your organization wouldn’t want to sponsor Django, he’d love to know why that is - understanding those blockers right now is almost as valuable as receiving actual cash. You can reach out to him at jacob at djangoproject.com.

Could we fund a Django LTS accessibility audit?

Django fellows and the Django Accessibility Team have been focusing significant effort on the accessibility of the Django admin. I found this very inspiring, and in combination with the talk of more funding for the foundation it put an idea in my head: what if every Django LTS release (once every two years) was backed by a full, professional accessibility audit, run by an agency staffed with developers who use screen readers?

Imagine how much impact it would have if the default Django admin interface had excellent, documented accessibility out of the box. It could improve things for hundreds of thousands of users, and set an excellent precedent for projects (and foundations) in the wider open source community.

This also feels to me like something that should be inherently attractive to sponsors. A lot of agencies use Django for government work, where accessibility is a requirement with teeth. Would one of those agencies like to be the “accessibility sponsor” for a major Django release?

Django fellows continue to provide outstanding value

The DSF’s fellowship program remains one of the most impactful initiatives I’ve seen anywhere for ensuring the ongoing sustainability of a community-driven open source project.

Both of the current fellows, Natalia Bidart and Sarah Boyce, were in attendance and gave talks. It was great getting to meet them in person.

If you’re not familiar with the program, the fellows are contractors who are paid by the DSF to keep the Django project ticking over - handling many of the somewhat less glamorous tasks of responsible open source maintenance such as ticket triage, release management, security fixes and code review.

The fellows program is in its tenth year, and is a key reason that Django continues to release new versions on a regular schedule despite having no single corporate parent with paid developers.

Unsurprisingly there is always more work than fellow capacity, hence Jacob’s desire to further expand the existing program.

The fellows program launched with a policy that fellows should not work on new feature development. I believe this was partly related to interpretation of IRS nonprofit guidelines which have since been reconsidered, and there is a growing consensus now that this policy should be dropped.

Django needs feature champions

Django has a well deserved reputation for stability, reliability and a dependable release process. It has less of a reputation for constantly turning out ground-breaking new features.

Long-time Django contributors who I talked to all had a similar position on this: the challenge here is that big new features need dedicated champions to both lead design and development on them and to push them through to completion.

The pool of community members who are both willing and able to take on these larger projects is currently too small.

There are a number of ways we could address this - most notably through investing financial resources in sponsoring feature development. This has worked well for Django in the past - Django’s migrations work was funded by a Kickstarter campaign back in 2013.

The Django Software Foundation will shortly be announcing details of elections for both the DSF board and the Django Steering Council. These are extremely influential positions for people who want to help solve some of these larger problems.

htmx fits Django really well

htmx is an incredibly good fit for the uncodified Django community philosophy of building for the web. It came up in multiple talks. It feels like it may be a solution that the Django community has been seeking for years, as a very compelling alternative to writing everything in SPA JavaScript and using Django purely as a backend via something like Django REST Framework.

I've been slightly resistant to embracing htmx myself purely because it's such a critical dependency and in the past I wasn't convinced of its staying power. It's now mature, stable and widely-enough used that I'm ready to consider it for my own long-term projects.

Django Ninja has positive buzz

I haven’t paid much attention to Django Ninja but it had a lot of very positive buzz at the conference as well, as a tool for quickly building full-featured, performative API endpoints (thanks to Rust-backed Pydantic for serialization) with interactive API docs powered by OpenAPI.

I respect Django REST Framework a lot, but my personal programming style leans away from Class Based Views, which it uses quite a bit. Django Ninja looks like it might fit my function-view biases better.

I wrote about Richard Terry’s excellent nanodjango single-file Django application tool the other day - Django Ninja comes baked into that project as well.

The three platinum sponsors for DjangoCon this year were REVSYS, Caktus Group and Valkey. Valkey were a late and somewhat surprising addition to the sponsorship lineup.

Valkey is the Linux Foundation backed fork of Redis, created in response to Redis ditching their Open Source license (which I took quite personally, having contributed my own free effort to promoting and improving Redis in the past).

Aside from expressing thanks to them, I usually don’t pay sponsors that much attention. For some reason this one hit differently - the fact that Valkey were ready to step in as a major sponsor despite being only a few months old has caused me to take that project a whole lot more seriously than I did before. I’ll certainly consider them next time I come across a Redis-shaped problem.

Durham has a world-class collection of tubas

My favorite category of Niche Museum is one that's available by appointment only where the person who collected everything is available to show you around.

I always check Atlas Obscura any time I visit a new city, and this time I was delighted to learn about The Vincent and Ethel Simonetti Historic Tuba Collection!

I promoted it in the DjangoCon US #outings Slack channel and got together a group of five conference attendees for a visit on Thursday, shortly before my flight.

It was peak Niche Museum. I’ve posted photos and notes over on my Niche Museums website, the first new article there in quite a while.

Tags: accessibility, conferences, django, djangocon, jacob-kaplan-moss, python, redis, dsf, pydantic, htmx

My @covidsewage bot now includes useful alt text

2024-08-25T16:09:49+00:00

My @covidsewage bot now includes useful alt text

I've been running a @covidsewage Mastodon bot for a while now, posting daily screenshots (taken with shot-scraper) of the Santa Clara County COVID in wastewater dashboard.

Prior to today the screenshot was accompanied by the decidedly unhelpful alt text "Screenshot of the latest Covid charts".

I finally fixed that today, closing issue #2 more than two years after I first opened it.

The screenshot is of a Microsoft Power BI dashboard. I hoped I could scrape the key information out of it using JavaScript, but the weirdness of their DOM proved insurmountable.

Instead, I'm using GPT-4o - specifically, this Python code (run using a python -c block in the GitHub Actions YAML file):

import base64, openai
client = openai.OpenAI()
with open('/tmp/covid.png', 'rb') as image_file:
    encoded_image = base64.b64encode(image_file.read()).decode('utf-8')
messages = [
    {'role': 'system',
     'content': 'Return the concentration levels in the sewersheds - single paragraph, no markdown'},
    {'role': 'user', 'content': [
        {'type': 'image_url', 'image_url': {
            'url': 'data:image/png;base64,' + encoded_image
        }}
    ]}
]
completion = client.chat.completions.create(model='gpt-4o', messages=messages)
print(completion.choices[0].message.content)

I'm base64 encoding the screenshot and sending it with this system prompt:

Return the concentration levels in the sewersheds - single paragraph, no markdown

Given this input image:

Here's the text that comes back:

The concentration levels of SARS-CoV-2 in the sewersheds from collected samples are as follows: San Jose Sewershed has a high concentration, Palo Alto Sewershed has a high concentration, Sunnyvale Sewershed has a high concentration, and Gilroy Sewershed has a medium concentration.

The full implementation can be found in the GitHub Actions workflow, which runs on a schedule at 7am Pacific time every day.

Tags: accessibility, alt-text, projects, ai, covid19, shot-scraper, openai, generative-ai, gpt-4, llms

Reckoning

2024-08-18T16:37:41+00:00

Reckoning

Alex Russell is a self-confessed Cassandra - doomed to speak truth that the wider Web industry stubbornly ignores. With this latest series of posts he is spitting fire.

The series is an "investigation into JavaScript-first frontend culture and how it broke US public services", in four parts.

In Part 2 — Object Lesson Alex profiles BenefitsCal, the California state portal for accessing SNAP food benefits (aka "food stamps"). On a 9Mbps connection, as can be expected in rural parts of California with populations most likely to need these services, the site takes 29.5 seconds to become usefully interactive, fetching more than 20MB of JavaScript (which isn't even correctly compressed) for a giant SPA that incoroprates React, Vue, the AWS JavaScript SDK, six user-agent parsing libraries and a whole lot more.

It doesn't have to be like this! GetCalFresh.org, the Code for America alternative to BenefitsCal, becomes interactive after 4 seconds. Despite not being the "official" site it has driven nearly half of all signups for California benefits.

The fundamental problem here is the Web industry's obsession with SPAs and JavaScript-first development - techniques that make sense for a tiny fraction of applications (Alex calls out document editors, chat and videoconferencing and maps, geospatial, and BI visualisations as apppropriate applications) but massively increase the cost and complexity for the vast majority of sites - especially sites primarily used on mobile and that shouldn't expect lengthy session times or multiple repeat visits.

There's so much great, quotable content in here. Don't miss out on the footnotes, like this one:

The JavaScript community's omertà regarding the consistent failure of frontend frameworks to deliver reasonable results at acceptable cost is likely to be remembered as one of the most shameful aspects of frontend's lost decade.

Had the risks been prominently signposted, dozens of teams I've worked with personally could have avoided months of painful remediation, and hundreds more sites I've traced could have avoided material revenue losses.

Too many engineering leaders have found their teams beached and unproductive for no reason other than the JavaScript community's dedication to a marketing-over-results ethos of toxic positivity.

In Part 4 — The Way Out Alex recommends the gov.uk Service Manual as a guide for building civic Web services that avoid these traps, thanks to the policy described in their Building a resilient frontend using progressive enhancement document.

Tags: accessibility, alex-russell, government, html, javascript, progressive-enhancement, web-performance, gov-uk

Serving a billion web requests with boring code

2024-06-28T16:22:45+00:00

Serving a billion web requests with boring code

Bill Mill provides a deep retrospective from his work helping build a relaunch of the medicare.gov/plan-compare site.

It's a fascinating case study of the choose boring technology mantra put into action. The "boring" choices here were PostgreSQL, Go and React, all three of which are so widely used and understood at this point that you're very unlikely to stumble into surprises with them.

Key goals for the site were accessibility, in terms of users, devices and performance. Despite best efforts:

The result fell prey after a few years to a common failure mode of react apps, and became quite heavy and loaded somewhat slowly.

I've seen this pattern myself many times over, and I'd love to understand why. React itself isn't a particularly large dependency but somehow it always seems to lead to architectural bloat over time. Maybe that's more of an SPA thing than something that's specific to React.

Loads of other interesting details in here. The ETL details - where brand new read-only RDS databases were spun up every morning after a four hour build process - are particularly notable.

Via Lobste.rs

Tags: accessibility, go, postgresql, scaling, react, boring-technology

Transcripts on Apple Podcasts

2024-06-13T20:14:19+00:00

Transcripts on Apple Podcasts

I missed this when it launched back in March: the Apple Podcasts app now features searchable transcripts, including the ability to tap on text and jump to that point in the audio.

Confusingly, you can only tap to navigate using the view of the transcript that comes up when you hit the quote mark icon during playback - if you click the Transcript link from the episode listing page you get a static transcript without the navigation option.

Transcripts are created automatically server-side by Apple, or podcast authors can upload their own edited transcript using Apple Podcasts Connect.

Via A few people on Twitter told me about this

Tags: accessibility, apple, podcasts, transcripts

Experimenting with local alt text generation in Firefox Nightly

2024-06-02T13:12:44+00:00

Experimenting with local alt text generation in Firefox Nightly

The PDF editor in Firefox (confession: I did not know Firefox ships with a PDF editor) is getting an experimental feature that can help suggest alt text for images for the human editor to then adapt and improve on.

This is a great application of AI, made all the more interesting here because Firefox will run a local model on-device for this, using a custom trained model they describe as "our 182M parameters model using a Distilled version of GPT-2 alongside a Vision Transformer (ViT) image encoder".

The model uses WebAssembly with ONNX running in Transfomers.js, and will be downloaded the first time the feature is put to use.

Via @mozhacks

Tags: accessibility, alt-text, firefox, javascript, mozilla, pdf, ai, webassembly, llms, transformers-js

Cally: Accessibility statement

2024-04-02T19:38:19+00:00

Cally: Accessibility statement

Cally is a neat new open source date (and date range) picker Web Component by Nick Williams.

It’s framework agnostic and weighs less than 9KB grilled, but the best feature is this detailed page of documentation covering its accessibility story, including how it was tested—in JAWS, NVDA and VoiceOver.

I’d love to see other open source JavaScript libraries follow this example.

Via @wickynilliams

Tags: accessibility, javascript, open-source, web-components

Guidepup

2024-03-14T04:07:49+00:00

Guidepup

I’ve been hoping to find something like this for years. Guidepup is “a screen reader driver for test automation”—you can use it to automate both VoiceOver on macOS and NVDA on Windows, and it can both drive the screen reader for automated tests and even produce a video at the end of the test.

Also available: @guidepup/playwright, providing integration with the Playwright browser automation testing framework.

I’d love to see open source JavaScript libraries both use something like this for their testing and publish videos of the tests to demonstrate how they work in these common screen readers.

Tags: accessibility, screen-readers, playwright

Ice Cubes GPT-4 prompts

2023-12-06T19:38:07+00:00

Ice Cubes GPT-4 prompts

The Ice Cubes open source Mastodon app recently grew a very good "describe this image" feature to help people add alt text to their images. I had a dig around in their repo and it turns out they're using GPT-4 Vision for this (and regular GPT-4 for other features), passing the image with this prompt:

What’s in this image? Be brief, it's for image alt description on a social network. Don't write in the first person.

Tags: accessibility, alt-text, ai, prompt-engineering, generative-ai, mastodon, gpt-4, llms, vision-llms

New Default: Underlined Links for Improved Accessibility (GitHub Blog)

2023-10-19T16:19:30+00:00

New Default: Underlined Links for Improved Accessibility (GitHub Blog)

“By default, links within text blocks on GitHub are now underlined. This ensures links are easily distinguishable from surrounding text.”

Tags: accessibility, design, github

You can stop using user-scalable=no and maximum-scale=1 in viewport meta tags now

2023-08-04T23:41:55+00:00

You can stop using user-scalable=no and maximum-scale=1 in viewport meta tags now

Luke Plant points out that your meta viewport tag should stick to just “width=device-width, initial-scale=1” these days—the user-scalable=no and maximum-scale=1 attributes are no longer necessary, and have a negative impact on accessibility, especially for Android users.

Tags: accessibility, html, luke-plant, mobile, mobileweb

The anatomy of visually-hidden

2023-02-11T00:37:28+00:00

The anatomy of visually-hidden

James Edwards provides a detailed breakdown of the current recommended CSS for hiding content while keeping it available for assistive technologies in the browser accessibility and render trees. Lots of accumulated tricks and screen reader special cases in this.

Via Ben Myers

Tags: accessibility, css, screen-readers

It looks like I'm moving to Mastodon

2022-11-05T05:32:20+00:00

Elon Musk laid off about half of Twitter this morning. There are many terrible stories emerging about how this went down, but one that particularly struck me was that he laid off the entire accessibility team. For me this feels like a microcosm of the whole situation. Twitter's priorities are no longer even remotely aligned with my own.

I've been using Twitter since November 2006 - wow, that's 16 years! I've accumulated 42,804 followers there. It's been really good to me, and I've invested a lot of work generating content there to feed the machine.

I can't see myself putting the same work in to help the world's (current) richest man pay the billion dollar annual interest on the loans he took out to buy the place on a weird narcissistic whim.

So I've started to explore Mastodon - and so far it's exceeding all of my expectations.

My new profile is at https://fedi.simonwillison.net/@simon - you can follow @simon@simonwillison.net in your Mastodon client of choice.

Not ready to sign up for Mastodon? It turns out RSS support is baked in too - you can subscribe to https://fedi.simonwillison.net/@simon.rss in your feed reader (I really like NetNewsWire for macOS and iOS these days).

Why Mastodon?

The lesson I have learned from Twitter is that, even if a service you trust makes it past an IPO and becomes a public company, there's always a risk that it can be bought by someone who very much doesn't share your values.

Mastodon has been designed to avoid this from the start. It operates as a federated network of independent servers, each of which is run by a different person or organization with the ability to set their own rules and standards.

You can also host your own instance on your own domain.

My initial nudge to try this out was from Jacob and Andrew, who figured out how to do exactly that:

The Fediverse, And Custom Domains - Andrew Godwin
Setting up a personal Fediverse ID / Mastodon instance - Jacob Kaplan-Moss

Andrew and Jacob both opted to pay masto.host to run their instance for them. I've decided to do the same. It's on my domain, which means if I ever want to run it myself I can do so without any visible disruption.

I'm paying $9/month. I find it darkly amusing that this is a dollar more than Elon has been planning to charge for users to keep their verified status on Twitter!

If you don't want to use your own domain there are plenty of good free options, though I recommend reading Ash Furrow's post about his shutdown of mastodon.technology to help understand how much of a commitment it is for the admins who run a free instance.

This post by @klillington@mastodon.ie has some good links for getting started understanding the system. I particularly enjoyed Nikodemus’ Guide to Mastodon as it matched most closely the questions I had at first.

Initial impressions

Despite taking the second hardest route to joining Mastodon (the hardest route is spinning up a new server from scratch) it took me just less than an hour to get started. I wrote up a TIL describing what I did - more or less directly following the steps described by Andrew and Jacob.

I signed into my new account and started following people, by pasting in their full Mastodon names (mine is @simon@simonwillison.net). I was initially surprised that this did nothing: your timeline won't be populated until the people you follow have said something.

And then people started to toot, and my timeline slowly kicked into life.

And it was really, really pleasant.

My fear was that everyone on Mastodon would spend all of their time talking about Mastodon - especially given the current news. And sure, there's some of that. (I'm obviously guilty here.)

But there's lots of stuff that isn't that. The 500 character limit gives people a bit more space, and replies work much like they do on Twitter. I followed a bunch of people, replied to a few things, posted some pelican photos and it all worked pretty much exactly as I hoped it would.

It's also attracting very much the kind of people I want to hang out with. Mastodon is, unsurprisingly, entirely populated by nerds. But the variety of nerds is highly pleasing to me.

I've been checking in on the #introduction hashtag and I'm seeing artists, academics, writers, historians. It's not just programmers. The variety of interest areas on Twitter is the thing I'll miss most about it, so seeing that start to become true on Mastodon too is a huge relief.

Considering how complicated a federated network is, the fact that it's this smooth to use is really impressive. It helps that they've had six years to iron out the wrinkles - the network seems to be coping with the massive influx of new users over the past few days really well.

I'm also appreciating how much thought has been put into the design of the system. Quote tweeting isn't supported, for reasons explained by Eugen Rochko in this 2018 post:

Another feature that has been requested almost since the start, and which I keep rejecting is quoting messages. Coming back to my disclaimer, of course it’s impossible to prevent people from sharing screenshots or linking to public resources, but quoting messages is immediately actionable. It makes it a lot easier for people to immediately engage with the quoted content… and it usually doesn’t lead to anything good. When people use quotes to reply to other people, conversations become performative power plays. “Heed, my followers, how I dunk on this fool!” When you use the reply function, your message is broadcast only to people who happen to follow you both. It means one person’s follower count doesn’t play a massive role in the conversation. A quote, on the other hand, very often invites the followers to join in on the conversation, and whoever has got more of them ends up having the upper hand and massively stressing out the other person.

Mastodon so far feels much more chilled out than Twitter. I get the impression this is by design. When there's no profit motive to "maximize engagement" you can design features to optimize for a different set of goals.

And there's an API

Unsurprisingly, Mastodon has a powerful API. It's necessary for the system itself to work - those toots aren't going to federate themselves!

Poking around with it is really fun.

First, a friendly note. @pamela@bsd.network wrote the following:

Hacky folks, please resist finding ways to scrape the fediverse, build archives, automate tools and connect to people via bot without their consent.

[...]

Whatever your thing is, make it 100% opt-in. Make it appropriate for a significantly more at-risk user than you are. Make sure it forgets things, purges info about servers it can't contact, can't operate in any sort of logged-in mode where consent is an issue.

We will straight up help advertise your cool thing if it respects users properly and takes the time to consider the safety and preferences of every person involved. There are a lot of fun, thoughtfully-designed toys! And there are a lot of people really tired of having to come and tell you off when you wanted to help, honestly. Help yourself and ask around before you flip on your cool new thing, let folks point out what you're missing.

(Read the whole thing, it's great.)

So far I've done a couple of things.

I built a Git scraper to track the list of peer instances that various servers have picked up. This feels like a reasonable piece of public information to track, and it's a fun way to get a feel for how the network is growing.

I also figured out how to Export a Mastodon timeline to SQLite using the timelines API and my paginate-json and sqlite-utils CLI tools, so I could explore it in Datasette.

Running my own instance means I have no ethical qualms at all about hammering away at my own API endpoint as fast as I like!

I like to follow a lot of different people, and I don't like to feel committed to reading everything that crosses my timeline - so I expect that the feature I'll miss most from Twitter will be the algorithmic timeline! This is very much not in the spirit of Mastodon, which is firmly committed to a reverse chronological sort order.

But with access to the raw data I can start experimenting with alternative timeline solutions myself.

I'm somewhat intrigued by the idea of iterating on my own algorithmic timeline, to try and keep the variety of content high while hopefully ensuring I'm most likely to catch the highlights (whatever that means.)

Past experience building recommendation systems has taught me that one of the smartest seeming things you can do is pick the top 100 most interesting looking things based on very loose criteria and then apply random.shuffle() to produce a final feed!

I have a hunch that this is going to be a lot of fun.

Tags: accessibility, twitter, mastodon, fediverse

WebAIM guide to using iOS VoiceOver to evaluate web accessibility

2022-04-24T19:35:59+00:00

WebAIM guide to using iOS VoiceOver to evaluate web accessibility

I asked for pointers on learning to use VoiceOver on my iPhone for accessibility testing today and Matt Hobbs pointed me to this tutorial from the WebAIM group at Utah State University.

Via @TheRealNooshu

Tags: accessibility, screen-readers, ios

Weeknotes: Distracted by Playwright

2022-03-12T00:30:26+00:00

My goal for this week was to unblock progress on Datasette by finally finishing the dash encoding implementation I described last week. I was getting close, and then I got very distracted by Playwright.

Dash encoding v2

In Why I invented “dash encoding”, a new encoding scheme for URL paths I described a new mechanism I had invented for handling the gnarly problem of including table names with / characters in the URL path on Datasette. The very short version: you can't use URL encoding in a path, because common proxies (including Apache and Nginx) will decode them before they get to your application.

Thanks to feedback on that post I actually changed my design: I'm now using a variant of percent encoding that uses the - instead of the %. More details in the issue - and I'll write this up fully once I've finished landing the change.

shot-scraper and Playwright

I thoroughly nerd-sniped myself with this one. I started investigating possibilities for automatically generating screeshots for documentation, and realized that Playwright made this substantially easier than it has been in the past.

The result was shot-scraper - a new command-line utility for taking screenshots of web pages, or portions of web pages - and for running through a set of screenshots defined in a YAML file.

I still can't quite believe how quickly this came together.

Every now and then a tool comes along which adds a fundamental new set of capabilities to your toolbox, and can be multiplied against other tools to open up a huge range of possibilities.

Playwright feels like one of those tools.

A quick pip install playwright is all it takes to start writing robust browser automation tools, using dedicated standalone headless instances of multiple browsers that are installed for you using playwright install.

It's easy to run in CI - getting it working in GitHub Actions was trivial.

shot-scraper is my first project built on Playwright, but there will definitely be more.

shot-scraper accessibility

I started a Twitter conversation asking for ways to write automated tests that exercise screen readers - not just running audit rules, but actually simulating what happens when a screen reader user attempts to navigate through a specific flow within an application.

The most interesting answer I had was from Ben Mustill-Rose, who built a system for automating tests against an Android screen reader while working on BBC iPlayer - demo here.

@fardarter pointed me back to Playwright again, which turns out to have an Accessibility snapshot mechanism that can dump out the current state of the Chromium accessibility tree.

I couldn't resist adding that to shot-scraper - so now you can run the following to see the accessibility tree for a web page:

~ % shot-scraper accessibility https://datasette.io
{
    "role": "WebArea",
    "name": "Datasette: An open source multi-tool for exploring and publishing data",
    "children": [
        {
            "role": "link",
            "name": "Uses"
        },
        {
            "role": "link",
            "name": "Documentation"
        },

Full output here.

As a really fun bonus trick: since the output is JSON, you can pipe it into sqlite-utils insert to get a SQLite database:

shot-scraper accessibility https://datasette.io \
    | jq .children | sqlite-utils insert \
    /tmp/accessibility.db nodes - --alter

And then open it in Datasette Desktop and start faceting by role and heading level!

sqlite-utils documentation improvements

I complained on Twitter that the way type information was displayed in the Sphinx sqlite-utils API reference documentation was ugly:

Adam Johnson pointed me to the autodoc_typehints = "description" option which fixes this. I spent a while tidying up the documentation to work better with this, mainly by adding a whole bunch of :param name: description tags that I had previously omitted. That work happenen in this issue. I think it looks much better now:

Releases this week

image-diff: 0.2.1 - (3 releases total) - 2022-03-11
CLI tool for comparing images
sqlite-utils: 3.25.1 - (98 releases total) - 2022-03-11
Python CLI utility and library for manipulating SQLite databases
shot-scraper: 0.4 - (5 releases total) - 2022-03-10
Automated website screenshots using GitHub Actions
django-sql-dashboard: 1.0.2 - (34 releases total) - 2022-03-08
Django app for building dashboards using raw SQL queries
geojson-to-sqlite: 1.0 - (8 releases total) - 2022-03-04
CLI tool for converting GeoJSON files to SQLite (with SpatiaLite)
xml-analyser: 1.3 - (4 releases total) - 2022-03-01
Simple command line tool for quickly analysing the structure of an arbitrary XML file
datasette-dateutil: 0.3 - (4 releases total) - 2022-03-01
dateutil functions for Datasette

TIL this week

Tags: accessibility, documentation, datasette, weeknotes, sphinx-docs, playwright, shot-scraper

A Complete Guide To Accessible Front-End Components

2021-03-23T01:06:04+00:00

A Complete Guide To Accessible Front-End Components

I’m so excited about this article: it brings together an absolute wealth of resources on accessible front-end components, including many existing component implementations that are accessible out of the box. Date pickers, autocomplete widgets, modals, menus—all sorts of things that I’ve been dragging my heels on implementing because I didn’t fully understand their accessibility implications.

Tags: accessibility, javascript

Using the HTML lang attribute

2019-04-18T21:09:31+00:00

Using the HTML lang attribute

TIL the HTML lang attribute is used by screen readers to understand how to provide the correct accent and pronunciation.

Via Lindsey Kopacz

Tags: accessibility, html, l10n, screen-readers

Quoting Bruce Lawson

2018-12-19T13:07:14+00:00

If you wrap your main content – that is, the stuff that isn’t navigation, logo and main header etc – in a
tag, a screen reader user can jump immediately to it using a keyboard shortcut. Imagine how useful that is – they don’t have to listen to all the content before it, or tab through it to get to the main meat of your page.

— Bruce Lawson

Tags: accessibility, bruce-lawson, html

How technology helped a blind athlete run free at the New York Marathon

2017-11-06T16:58:57+00:00

How technology helped a blind athlete run free at the New York Marathon

Fascinating piece on technology to help blind people better navigate the world—combing GPS and chest-mounted ultrasonic sonar.

Tags: accessibility

Quoting Daniel Göransson

2017-11-06T16:56:59+00:00

Skip the title text! Nobody uses them – they don’t work on touch screens and on desktop they require that the user hovers for a while over an image, which nobody does. Also, adding a title-text makes some screen readers both read the title-text and the alt-text, which becomes redundant.

— Daniel Göransson

Tags: accessibility, alt-text

Alt-texts: The Ultimate Guide

2017-11-06T16:54:19+00:00

Alt-texts: The Ultimate Guide

By Daniel Göransson, a web developer with vision impairment who uses a screen reader. This is the best, most practical guide to writing image alt text I’ve seen. Just one of the neat tips contained within: consider ending your alt text in a period, so the screen user knows to pause.

Tags: accessibility, alt-text

My First Week with the iPhone

2010-10-03T12:20:00+00:00

My First Week with the iPhone

A blind user describes the experience of using VoiceOver on the iPhone, including the joy of discovering the Color Identifier app which speaks the names of colours picked up by the iPhone’s camera. “ I used color cues to find my pumpkin plants, by looking for the green among the brown and stone. I spent ten minutes looking at my pumpkin plants, with their leaves of green and lemon-ginger.”

Tags: accessibility, iphone, recovered

Accessibility and Degradation in Cappuccino

2009-02-26T21:39:33+00:00

Accessibility and Degradation in Cappuccino

Ross Boucher from 280 North responds to Drew McLellan.

Tags: 280north, accessibility, cappuccino, drew-mclellan, javascript, ross-boucher

The Cost of Accessibility

2009-02-25T22:31:47+00:00

The Cost of Accessibility

Drew McLellan comments on the seemingly inevitable march towards JavaScript dependent applications, and argues that JavaScript frameworks such as Cappuccino have a duty to integrate accessibility in to their core.

Tags: accessibility, cappuccino, drew-mclellan, javascript

Simon Willison's Weblog: accessibility

Introducing ChatGPT Atlas

Should form labels be wrapped or separate?

New dashboard: alt text for all my images

Notes from my Accessibility and Gen AI podcast appearance

LLMs for drafting alt text

Is it ethical to build unreliable accessibility tools?

Why I don't feel threatened as a software engineer

Why are my live regions not working?

Clay UI library

Themes from DjangoCon US 2024

Growing the Django Software Foundation (DSF)

Could we fund a Django LTS accessibility audit?

Django fellows continue to provide outstanding value

Django needs feature champions

htmx fits Django really well

Django Ninja has positive buzz

Valkey as a last-minute sponsor

Durham has a world-class collection of tubas

My @covidsewage bot now includes useful alt text

Reckoning

Serving a billion web requests with boring code

Transcripts on Apple Podcasts

Experimenting with local alt text generation in Firefox Nightly

Cally: Accessibility statement

Guidepup

Ice Cubes GPT-4 prompts

New Default: Underlined Links for Improved Accessibility (GitHub Blog)

You can stop using user-scalable=no and maximum-scale=1 in viewport meta tags now

The anatomy of visually-hidden

It looks like I'm moving to Mastodon

Why Mastodon?

Initial impressions

And there's an API

WebAIM guide to using iOS VoiceOver to evaluate web accessibility

Weeknotes: Distracted by Playwright

Dash encoding v2

shot-scraper and Playwright

shot-scraper accessibility

sqlite-utils documentation improvements

Releases this week

TIL this week

A Complete Guide To Accessible Front-End Components

Using the HTML lang attribute

Quoting Bruce Lawson

How technology helped a blind athlete run free at the New York Marathon

Quoting Daniel Göransson

Alt-texts: The Ultimate Guide

My First Week with the iPhone

Accessibility and Degradation in Cappuccino

The Cost of Accessibility