Why agents reach your site and still fail

Almost everything written about agent readiness, including most of this library, is about two questions: can a machine reach your pages, and can it understand them. There is a third question that decides whether any of it turns into money, and it is the one nobody instruments. Once an agent is on your site, with a real buyer waiting behind it, can it actually finish?

Reviewed September 23, 2026 · 11 min read

When we scanned 652 small-business websites for the agent readiness study, a site counted as invisible if its homepage did not return successfully, or if it offered no parseable structured data and none of the six machine-readable artifacts we probed for. That is a defensible definition and it measures something real. It also stops one step short of the thing that pays.

A site can return a clean 200, carry immaculate JSON-LD, and still defeat an agent that is trying to book a table or buy a jacket, because reading and doing are different problems. The agent arrives, understands you perfectly, and then hits a consent wall, or a modal, or a checkout step that assumes a human is holding the mouse. The buyer behind it gets a shrug and goes somewhere easier.

Reachable, readable, operable#

Three gates, in order, and each one is invisible to the instruments that measure the others.

The reason for the imbalance is measurement. The first two gates fail loudly and leave evidence: a 403 in your logs, a validator error, a rich result that never appears. The third fails silently. There is no log line for an agent that got to step four of your checkout and gave up, and no support ticket either, because the person it was working for never knew your name.

The agent is a browser with a supervisor#

It helps to be concrete about what is actually visiting. OpenAI describes its cloud browser as having its own browser on a separate computer, able to read web pages, click buttons, enter information into forms, and carry out steps on supported public and signed-in websites. So it is not a crawler fetching HTML. It is closer to a careful, literal-minded temp who has been told to get something done and will not improvise.

That temp has two ways of stopping, and the difference matters. It pauses when it needs your input, a sign-in, or a confirmation, which returns control to the person and is often the correct behaviour. And it hands the task back when it cannot proceed, which OpenAI states plainly: support varies by website and action, and you may need to take over or complete the final step yourself.

The obstacle course, in the order an agent meets it#

Everything below is ordinary web design. None of it was built to obstruct anyone. All of it was designed for a human with a pointer, a sense of impatience, and the ability to guess what a button means.

Notice that none of these are bugs. Every one of them shipped for a reason a sensible person would defend, which is exactly why they survive. They were all designed against an assumption, that the visitor is a person, which quietly stopped being universally true.

Your bot defences are now also your customer defences#

OpenAI states it without editorialising: some websites use security measures that restrict access from automated browser agents. As a sentence about the web that is neutral. As a sentence about your revenue it is expensive, because the measure that stops a scraper and the measure that stops a customer’s agent are frequently the same measure.

The old answer was to pick a side, block aggressively and lose the good traffic, or allow broadly and eat the abuse. That answer is out of date. Signed requests now let an origin tell a named, cryptographically identified agent apart from an anonymous one, which turns a binary into a policy. How to verify an AI agent covers the mechanism; the point here is simply that you no longer have to choose blindly, and a bot rule written before that changed is a rule worth rereading.

What your server says when it says no#

Sometimes refusing is correct. You are under load, or the request really is abusive, or the thing is gone. Refusing is not the problem. Refusing uninformatively is, because status codes are the only vocabulary you share with something that will never read your error page.

The last row is the common one and it is the worst, because it is the only failure that also poisons your data. An agent that receives a successful response containing the words "something went wrong" has no way to know that is not your page, and may cache it, quote it, or treat the absence of a price as a fact about your pricing.

The missing half of a good refusal is Retry-After, which is a single header carrying either a number of seconds or an HTTP date, and which is used with 503, 429 and 301. A bare 429 tells an agent it went too fast and nothing about when it may return, so it guesses, and its guess is either too soon, which looks like abuse, or too late, which is a sale that happened somewhere else. One header removes the guess.

Make the commitment point obvious, because the agent is looking for it#

This is the part that is genuinely about design rather than plumbing. OpenAI says its agent is built to request confirmation before actions that could be hard to reverse or create a financial, legal, account, or other real-world commitment. That is a sensible safety property, and it is also a structural assumption about your site: that there is a recognisable moment where commitment happens.

Flows that have one clear, well-labelled commit step work with that assumption. Flows that scatter commitment, that charge on a screen which looks like navigation, that bury the irreversible action behind a button labelled Continue, fight it. The agent either stops too early, and the buyer is asked to take over for no reason, or it cannot identify the moment at all and stops short.

The instruction that follows is unfashionable: label the irreversible thing as irreversible. A button that says Place order and charge my card is worse marketing copy and a better commit signal, for the machine and, as it happens, for the human being represented by it.

Twenty minutes to find out where you stand#

1. Fetch your own product or booking page with curl and read the raw HTML. If the price, the availability or the button text is not in that output, it does not exist for anything that does not run a browser.
2. Open the same page in a browser with JavaScript disabled. This is the crudest possible proxy for the readable gate and it takes thirty seconds.
3. Now do the operable test, which nobody does: open a private window, go to your own site, and complete a real purchase or booking without using any knowledge you have as the owner. Count every interruption. Each one is a place an agent can stop.
4. Ask an actual assistant to do the same task on your site, and watch where it hands back. The handback point is your answer, and it is usually not where you expected.
5. Trigger a rate limit in staging and read the response headers. Check for Retry-After, and check that your failures are not 200s carrying apologies.
6. Grep your logs for 403s and 429s served to named agents, which measuring AI agent traffic covers properly. A wall of them means the first gate is failing and the other two never got a turn.

The third item is the one people skip and it is the one that finds things. Owners cannot fail their own checkout, because they know which button is which, which field is optional and what the confusing step actually means. An agent knows none of that. It reads exactly what is written and does exactly what the interface implies, which makes it an unusually honest usability tester that costs nothing and never gets bored.

None of this is a structured data problem, which is why it survives in businesses that have done the structured data work properly. Being found is a solved discipline with tools and validators. Being finishable is not, and for now the merchants who take the twenty minutes are competing against a field that has not thought about it at all.