Last time I wrote about the rule who wasn’t there – the dead regulation an entire industry keeps citing because the summary is easier to live with than the source. The moral was so simple it fit in a sentence: stop trusting the summary; go meet the source. I believed it when I wrote it. I believe it now. I have spent the past two weeks living by it as strictly as I ever have in my life, reading recovered primary documents – regulator guidelines, court opinions, the actual papers behind the numbers everyone quotes – with an AI collaborator who has been trained, by me, at some length, to treat secondhand claims the way a food inspector treats a warm prawn buffet.
So this is the confession the sermon earned: we went to meet the source, exactly like I keep telling everyone to. And the source, when we got there, turned out to be lying to one of us.
Here’s the example, and it’s the best thing I’ve learned all year. The European Data Protection Board publishes the EU’s actual rulebook on deceptive design – Guidelines 03/2022, the document that tells companies when highlighting one button over another crosses the line. It’s the primary source’s primary source. If you cite anything in this field, you cite this, and you cite it by paragraph, because that’s how lawyers point at things. Paragraph 57 holds the sentence the whole consent-banner argument turns on.
Try to find paragraph 57 by machine and something quietly impossible happens: the paragraph numbers are not in the text. Extract the PDF with the standard tools – we tried the usual two, the ones running inside a thousand research pipelines right now – and the numbers simply aren’t there. Not garbled. Absent. The document’s own footnotes refer to “para. 79-81,” so the numbers must exist somewhere; the text layer just doesn’t contain them. They live in the rendered margin, drawn where eyes can find them, invisible to anything reading the document as text.
The fix, when we finally found it, is the punchline of this entire essay. We rendered the page as an image – turned the PDF back into a picture – and my AI looked at it. Looked at it the way a sighted human reads a printed page. And there they were, 56, 57, 58, marching down the margin, visible the whole time to anyone reading with vision.
I need you to understand what my job is, so you understand what happened in my chest at that moment. I have spent my career saying one sentence in a hundred different rooms: if information exists only in the visual presentation, some of your readers don’t get it. It’s the oldest rule in accessibility. It’s why alt text exists, why headings have to be real headings and not just big bold text, why WCAG has a criterion – 1.3.1, Info and Relationships – that says structure conveyed visually has to exist programmatically too. Big bold text looks like a heading to eyes and is nothing to a screen reader. The EDPB’s paragraph numbers look like paragraph numbers to eyes and are nothing – literally nothing, zero characters – to every machine that tries to cite them from the text.
The body that writes Europe’s rules against deceptive interfaces has shipped its rulebook with information conveyed only in presentation. The regulators’ documents fail the regulators’ own tests. I want to be delighted about this, and I am, in the way you’re delighted by a fire station burning down. But mostly it makes me want to grab people by the lapels, because it isn’t one document and it isn’t one trap – and each one we found sits a little deeper in the record than the last.
It escalates. The same guidelines cite their own annex wrong. The body of the document refers to checklist items 4.6.2 and 4.6.3; the annex itself numbers those items 4.6.1 and 4.6.2. Somewhere between version one and version two, an item moved and the cross-references didn’t. Cite the document’s own text faithfully and you will point your reader at the wrong entry in the document’s own appendix. The trap isn’t sloppiness in some blogger’s summary. The trap is in the primary source, waiting specifically for the people diligent enough to quote it exactly.
Then it reaches the canon. The academic paper this whole field is built on – the one every detector, dataset, and law review article cites – carries its central claim in a table whose coding symbols (filled circle for “always,” empty for “never”) extract as pure garbage. The claim everyone repeats from that table can only be verified by looking at the rendered page. We looked; it’s true; but the only reason we know is that we stopped reading like a machine.
Then the machines stop merely inheriting errors and start minting their own. An AI summarizer, asked about two European Commission press releases, swapped the reference numbers between them – attributed each document’s identity to the other, smoothly, confidently, in a way no human reading the PDFs side by side would do. (A different AI caught it by reading the PDFs side by side. There’s a whole other essay in that sentence. Sigh.)
And then the fifth one aimed at me and missed, which is my favorite, because the reason it missed doesn’t scale. A peer-reviewed paper that measures whether cookie banners are readable names its metric, in its one defining sentence, the “Flesch-Kincaid (FK) reading ease test.” That is two different instruments welded at the hyphen: the Flesch-Kincaid grade level and the Flesch Reading Ease score. The formula printed three lines below, and the 0-to-100 scale, belong entirely to the second. Quote it faithfully, compress the chimera to its famous half the way every reader silently does, and you will describe a 0-to-100 score using the name of a scale that tops out around 18. I didn’t catch it by checking the citation; the citation is no help, since both metrics trace to the same 1975 Navy report, so the reference checks out no matter which name you believe. I caught it because I have used Flesch-Kincaid constantly for years, and “a score of 50” read like a shoe size of 50: not impossible, exactly, just wrong enough to stop me mid-sentence. I had never even heard of Flesch Reading Ease until this week. I didn’t know what the right answer was. I only knew the number wasn’t one my instrument says. That is the whole defense: not a habit, not a checklist, just mileage. The machines have a different defense, and it half works. One AI, reading the same paper, inherited the welded name so faithfully that it wrote it into the bullet points for this essay. Another flagged the mismatch immediately, because it knows both formulas and noticed the paper’s arithmetic belonged to the other one. Same species, opposite outcomes, and no way to know in advance which reading you have been handed. My alarm fired because I had worn the tool smooth; the machine’s alarm fired because it had read everything, once, thinly. Neither reader is reliable alone. The thing that was reliable, the only thing in this whole story that was, is that there were three of us, and we disagreed.
Five traps. Four of them share a shape: the truth is in the layer presented to eyes, and the error is in the layer handed to machines. The fifth is worse. The error is in the words themselves, in both layers, and the only thing that catches it is a reader who has personally worn the instrument smooth. Not one of the five lives in a summary, a hot take, or a vendor webinar. Every single one lives in the primary record.
Now, you could file this under “PDFs are bad,” shrug, and go about your day, if it weren’t for who is doing the reading now. Compliance crawlers read the record. Research pipelines read the record. The AI agents everyone is breathlessly deploying read the record – and they read it precisely the way that loses the numbers. The European Commission, in the evidence base for its next big consumer law, frets that principle-based rules resist “automated compliance checks through webcrawlers.” Chrome now publishes guidance called “Accessibility for agents.” The machines are becoming the record’s most numerous readers, and the record speaks to them in a voice that drops the numbers, swaps the references, and misnames the instruments. And here is the part the fifth trap taught me: it isn’t that machines can’t catch these errors. One demonstrably did. It’s that pipelines run one reading, once, and ship whatever that single pass produced, and every human downstream inherits it. Nobody is deploying a quorum to parse consent banners. The machines read alone.
We have been here before. Every accessibility person has. A reader arrives that consumes structure instead of pixels; the documents turn out to have structure that exists only in the pixels; and the people who published them insist they’re fine, because they look fine, because everyone the publisher can imagine reads with eyes. Thirty years of screen-reader users know exactly how this movie goes, and how long the fixes take, and how many “it renders correctly” tickets get closed unresolved along the way. The only new thing is who has joined the excluded readers. It’s the machines now. The machines, and everyone downstream of them – which, increasingly, is all of us.
So the discipline survives, but it needs a corollary I didn’t have last month. Raw thing or nothing still stands – nothing in this story rehabilitates the summary. But the extracted text we treat as the document is itself a summary: a lossy account of what the page actually shows, often maintained by no one and checked by no one, quietly missing whatever the rendering kept in the margin. Sometimes meeting the source means looking at it the way a human does – with eyes, at the rendered page, the same way you’d verify a heading is really a heading. My collaborator found paragraph 57 by doing, mechanically, the thing I’ve begged developers to remember that some readers can’t do. The irony is palpable: the reader that could always fall back on eyes is the machine we keep saying doesn’t have any, and the readers who never could are the people whose exclusion taught us – long ago, in detail, at their expense – everything we needed to know to see this coming.
(Yes, the title of this essay is wrong twice. They’re paragraph numbers, not page numbers – and “only humans can see” isn’t even true, since a machine did eventually see them and a screen-reader user still can’t. I kept it anyway. A headline that doesn’t survive inspection of its own primary text felt like the most on-theme thing ever.)
The numbers were there the whole time, in the layer that only speaks to eyes. Until the record learns to speak to all of its readers… do the old thing, the boring thing: go and look.
Receipts (same rule as last time – an essay about the record had better cite the record):
The rulebook: EDPB, Guidelines 03/2022 on deceptive design patterns in social media platform interfaces, v2.0 (adopted 14 February 2023). Paragraph 57 is on page 24 – you’ll need eyes.
The extraction failure: reproducible with pypdf and pdftotext against the published PDF; the numbers exist only in the rendered layer. The document’s own footnotes (e.g., “para. 79-81”) confirm the numbering exists.
The mis-cross-referenced annex: compare the body’s “Annex I checklist 4.6.2 / 4.6.3” references with Annex section 4.6 (page 70), which numbers the same items 4.6.1 and 4.6.2.
The table only eyes can verify: Mathur et al., “Dark Patterns at Scale” (CSCW 2019), Table 1, page 81:12.
The welded name: Kirkman, Vaniea & Woods, “DarkDialogs” (EuroS&P 2023), the “Flesch-Kincaid (FK) reading ease test” defined on page 7. The printed formula and the 0-to-100 scale are Flesch Reading Ease; reference [38] (Kincaid et al., 1975) is the common origin of both metrics, which is how the names got fused in the first place.
The swapped press releases: European Commission IP/24/2227 (proceedings opened against TikTok over TikTok Lite, 22 April 2024) and IP/24/4161 (TikTok’s withdrawal commitments made binding, 5 August 2024). The summarization layer of a web-fetch tool attributed each release’s identity to the other; the correction came from reading both PDFs side by side.
The rule the record breaks: WCAG 2.2, Success Criterion 1.3.1, Info and Relationships.
The machines doing the reading: Chrome for Developers, “Accessibility for agents”; European Commission, SWD(2024) 230 final, on principle-based rules and “automated compliance checks through webcrawlers.”