Prompt to Page 0.7.2: three screen readers, and what an agent's test is worth

When 0.7.0 shipped, its accessibility statement said the fixes had been checked "with automated tests and by keyboard — not yet with a screen reader". I said then that a statement claiming conformance it hasn't verified is worse than none. Version 0.7.2, out now for macOS, is the release where screen readers arrived. With a large asterisk, and the asterisk is what this post is about.

An AI agent ran every one of those tests. Not a person.

How an agent tests with a screen reader

The testing round behind 0.7.2 covered:

• a second test with NVDA on Windows;
• a first test with JAWS;
• a first pass with VoiceOver on the Mac;
• the real Night sky and Desert Contrast themes;
• Windows text size at 200%, with the app's own zoom stacked on top, up to an effective 400%;
• the Windows UI Automation tree.

Each sitting followed a written script of journeys: set up the app, generate a page, edit a section, export. The agent pressed real keys through the operating system and recorded what the screen reader said.

How it "heard" depends on the screen reader, and the details are a little absurd:

• NVDA writes what it speaks to a log.
• JAWS keeps a speech history, and the free demo stops after 40 minutes, so its test ran in three 40-minute blocks.
• VoiceOver wouldn't take instructions from the agent at all. So the agent turned on VoiceOver's caption panel, the strip that shows what it is saying, and read it off the screen.

The numbers, on 0.7.1:

The VoiceOver pass ran on a 0.7.2 release candidate. 0.7.2 fixes what these tests found, apart from a short list the statement names.

What it found

JAWS couldn't hear the question. Every "Are you sure?" in the app opened in the operating system's own dialog. JAWS read it as a lone button: "Create an equivalent, button". The question itself was never spoken. The dialog also started on the button that goes ahead, and ignored Windows text size. In 0.7.2, every question opens in the app's own dialog. It starts on Cancel, closes with Escape, is read in full, and follows the zoom and text size.

The same dialog needs a different answer on each platform. Moving the questions into the app's dialog fixed JAWS. VoiceOver then read only the dialog's title and its Cancel button: Delete project never said which project. The fix for VoiceOver is to put the question in the dialog's name. But NVDA says a dialog's name twice, so on Windows that fix reads every question twice. So the Mac puts the question in the name, and Windows keeps it as the description. I had assumed there was one correct way to label a dialog. There turned out to be one per screen reader.

VoiceOver was slower than the notification. After you delete a section, a notification offers Undo and says to press F6 to reach it. VoiceOver reads the section that takes focus first, so "Press F6 to reach it" finished about 9 seconds later. The notification closed at 10. A notification with an action now stays for at least 20 seconds.

Focus kept falling to the top of the window. It happened whenever the control you were on went away: Stop when a generation ends, Apply in the HTML editor, Skip tour, Generate page giving way to Stop, and half a dozen more. A screen reader then starts again from the top. Each one now hands focus somewhere sensible, usually the chat box.

At 400% there was no room for the preview. Windows text size at 200%, with the interface zoomed to 200%, left the preview 4 pixels tall. Its buttons showed through a thin strip. The middle of the window now scrolls instead, so the preview keeps a usable height.

There's more in the changelog. The app now follows the Mac's Increase contrast and Reduce transparency settings. Text shows in full with wider letter and line spacing. Small buttons are at least 24 pixels square. The macOS menu bar finally has the app's commands in it.

What an agent's test can't tell me

An agent can tell me what was said and where focus went. That is worth a lot: every finding above is real, reproducible and now fixed.

What it can't tell me is whether the app is usable with a screen reader. Whether an hour of it is tiring. Whether the announcements are the right length, or just correct. Whether someone who has used JAWS for fifteen years would move around it the way my test script assumed, or reach for a shortcut the script never tried.

So the statement still says partially conformant, and it says why. Every test was run by an AI agent. No person who uses a screen reader has tested the app yet. Neither have any disabled people, and nobody has tried it with Voice Control, Voice Access or Dragon. I'd rather the statement say exactly what was done than imply a test that didn't happen.

If you use a screen reader, magnification or voice control and would test Prompt to Page for an hour, you are exactly who I need. Email hello@courtneyallen.co.uk.

The release held itself

One more thing, because it's the part I'm most glad about. Before publishing, an agent drives the installed build through a smoke test of everything new in the release.

The first 0.7.2 candidate failed it. The Prototype Kit export, the hand-off for teams who build in the GOV.UK Prototype Kit, was quietly dropping questions. The cause was a page where a name field and a date of birth shared a fieldset. The export turned that fieldset into a single date question: the name field vanished, and its hint appeared under the date. The bug came from a fix in this same release, the one the changelog calls "Prototype Kit pages keep their structure". 0.7.1 didn't have it.

The same smoke test found three sentences in my own changelog that weren't true as written:

• The new Japanese check for unhelpful error wording missed 「正しく入力してください」 ("please enter it correctly"), one of the commonest forms of exactly that wording.
• A new check for missing JavaScript hooks didn't recognise conditional questions wired in the most ordinary way.
• Dismiss on an error banner still dropped focus to the top of the window, after the changelog's first paragraph said that no longer happened.

So the release was held. Each problem was fixed with a test that failed first, the app was rebuilt, and the new build passed all 17 rows of a fresh smoke test. That's the build you can download.

Testing the claims, not just the code, keeps paying for itself.

Also in 0.7.2

Digital Agency Design System pages changed shape:

• a banner that opens a page carries its only <h1>;
• check answers keeps change links out of the answers;
• a date of birth is asked as year, month and day, and played back as one date (1990年4月1日);
• question pages ask one thing each;
• five new checks warn when a page drifts back to the old patterns.

New Japanese wording is proposed, pending review by a native speaker.

More answers play back on check answers: dates of birth, US-order dates on USWDS pages (11/23/1937), and addresses asked in parts, in the preview and in exported prototypes.

A confession. On macOS, the app's stored data grew by about a gigabyte for every hour it was open, even sitting idle, until the next launch cleared it. A check on the local AI ran every two seconds and rewrote its whole history each time. It now records only changes. Working in the app still adds some, about 10 to 30 MB an hour in testing, and the next launch clears it.

Getting it

Prompt to Page 0.7.2 is live for macOS (Apple Silicon) and Windows: download it, or accept the in-app update.

The changelog has every fix, and the accessibility statement has what was tested and what wasn't.

---

Prompt to Page is an independent project by Courtney Allen. It is not affiliated with, endorsed by, or connected to the Government Digital Service, the NHS, the U.S. General Services Administration, the Digital Agency (Japan), the Crown, or any government body. Design systems are referenced descriptively; protected marks and restricted fonts are never bundled.