What a slow page costs you with an AI agent

A shopper asks whether your jacket comes in medium and can arrive by Friday. Your product page answers quickly, but your shipping page stalls. An assistant now has to retry, look elsewhere or tell the shopper it cannot confirm delivery. That missing answer is the business cost of a slow page, even when the rest of your site is fast.

Reviewed October 1, 2026 · 9 min read

An agent needs more than permission to visit your site. It needs the answer while the task is still running. A page can return a perfectly valid response and still arrive too late to help the shopper.

Our guide to what stops agents completing tasks covers explicit failures, including status codes and retry instructions. Slow responses create a related problem: your server may still be working after the client has stopped waiting.

Your page can fail while the conversation continues#

A failed fetch does not necessarily end an assistant’s work. Anthropic’s web fetch documentation, for example, says Claude receives the error and continues the turn. That documents how one tool handles failure; it does not tell us what every assistant will say next.

The practical risk is that your current information is missing when the assistant makes its recommendation. It might try again, consult another source, use information it already has or explain that it cannot verify the answer. A wrong answer is possible, but it is not inevitable.

When an AI gets your business wrong explains why the source of an error matters. A timeout is a retrieval failure you can investigate and reduce. It does not turn every future answer into something beyond your control.

Deadlines matter before you hit them#

A request deadline creates a sharp boundary: once it expires, that attempt can fail. But time spent below the deadline still matters. It keeps the customer waiting and leaves less time for the assistant to check another product, confirm delivery or recover from an error.

There is no single timeout for AI traffic. A browser, fetch tool, proxy and overall task can each impose a limit. Some are configurable: the Model Context Protocol specification recommends request timeouts and says SDKs should allow them to be set per request. Those limits describe MCP calls, not a universal deadline for reading a web page.

For an integration you control, test against its actual limits. For unfamiliar visitors, reduce slow responses and leave room for variation instead of guessing at one safe cutoff.

Use 499 logs as a clue, then investigate#

If your server uses nginx, look for 499 entries in its access logs. nginx defines 499 for a client closing the connection while the request is being processed, before nginx tries to send the response header. The IANA registry leaves 499 unassigned; this is a logging convention, not a standard HTTP response delivered to the visitor.

For nginx’s standard combined log format, this command ranks URLs with 499 entries whose user-agent field names a user-triggered AI fetcher. Adapt the fields if you use a custom format.

# Candidate disconnects from named user-triggered fetchers
awk -F'"' '
  $3 ~ /^ 499 / &&
  $6 ~ /(ChatGPT-User|Claude-User|Perplexity-User)/ {
    split($2, request, " ")
    print request[2]
  }
' access.log | sort | uniq -c | sort -rn | head -20

The names come from the providers’ documentation for ChatGPT-User, Claude-User and Perplexity-User. Search crawlers and training crawlers have different jobs, so examine them separately. This filter is a starting point, not a complete inventory of agents.

Ranked counts show where to start, but busy URLs naturally collect more failures. Compare disconnects with total requests for the same URL and client group, and record request duration. Other stacks may use different codes or log fields. An empty 499 report does not prove that every request succeeded. Measuring AI agent traffic covers the broader approach.

Measure the whole task, including the shipping page#

One shopping question can require several requests. Checking a jacket’s size, stock and delivery date might involve a category page, three product pages, shipping information and a returns policy. That is an example of a possible path, not a fixed pattern every agent follows.

If six requests run one after another and each takes 600 milliseconds to complete, the requests alone take 3.6 seconds. Parallel requests can overlap, while redirects, retries and processing can add more time. Multiplying a site-wide median by the number of pages will not reliably predict the result.

Look for the slow request that holds up the next decision. A delayed shipping page matters when delivery is the shopper’s deciding factor, even if every product page is quick. Compare typical response times with the slow end of the distribution, such as the 95th percentile, and test the pages needed to complete real customer tasks.

Google also adjusts how much it can crawl#

Google’s crawl-budget documentation describes a separate effect: server responsiveness influences crawl capacity. Healthy, stable responses can allow more crawling; slower responses, 5xx errors and 429 rate limits can reduce it. Actual crawling also depends on demand.

This is evidence about Google’s crawling systems, not a rule shared by all AI assistants. Google also says detailed crawl-budget management is mainly relevant to large or frequently changing sites. Faster responses can help crawlers keep up, but more crawling does not guarantee indexing, rankings or AI citations.

Measure the first byte and the useful content#

web.dev’s TTFB guidance treats 0.8 seconds or less as good and over 1.8 seconds as poor. These are rough browser-performance benchmarks, not AI timeout limits. Navigation TTFB includes redirects, connection setup and the wait for a response; check what your measurement tool includes.

A first byte is not a finished answer. Headers or an early response can arrive before the content an agent needs. Measure complete responses and check when prices, stock and policies become readable. A fast empty shell can still leave a fetcher with nothing useful.

What to fix first#

1. Choose a real customer task, such as checking stock and delivery for a product. List the pages or endpoints needed to answer it.
2. Measure typical and slow responses on that path. Track first-byte time, complete-response time and whether the required facts are present.
3. Investigate disconnects and timeouts using request timings, verified client information and logs from each layer. Compare failure rates as well as raw counts.
4. Fix the slow dependencies that prevent the next step. Remove unnecessary redirects and update old links where you control them.
5. Include essential facts in the initial HTML or a documented API response. Anthropic’s web fetch tool, for example, does not support content that requires JavaScript rendering; a browser-based agent may behave differently.
6. Cache stable public information with a way to refresh it when policies change. Treat stock and prices separately so faster delivery does not mean stale answers.
7. Repeat the task after each change. Check that the answer arrives sooner and remains correct, including under the conditions that produced slow responses.

Start with the page that keeps a customer’s question from being answered. Often, the useful improvement is a straightforward one: a shipping policy that loads promptly, stock information in the response, or a redirect removed from a product link. Those fixes help both people and the assistants working for them.