Agent readiness

How to measure AI agent traffic (when your analytics cannot see it)

Google Analytics reports zero AI crawler visits no matter how many you get, because crawlers do not run JavaScript. That is one of three separate blind spots, each needing a different instrument. Here is the full measurement stack, with the user agents, the queries, and the verification step most guides skip.

Updated August 16, 2026 · 11 min read

If you open Google Analytics looking for AI crawler traffic, you will find none, and that tells you nothing about whether you have any. JavaScript-based analytics work by running a script in a browser; crawlers do not execute JavaScript, so they never fire the tag. Your site can be fetched thousands of times a month by GPTBot, ClaudeBot, and PerplexityBot while every dashboard you own reports a clean zero. The traffic is real and it is in your server logs, which is a place most marketing teams have never looked.

That is only the first blind spot. AI activity actually reaches your business through three separate layers, and each is invisible to a different tool: crawler fetches (visible only in server logs), referral visits from AI interfaces (visible in analytics, but under-attributed), and agent-completed transactions (visible only in your order data, since they generate no session at all). Measuring one and calling it done is how businesses conclude the channel is small while it quietly compounds.

This guide covers all three layers: what to look for, how to query it, which user agents matter and what each one actually does, how to verify a bot is who it claims to be, and what to change once you can finally see the numbers.

The three layers, and what sees each one

The third row is the one that surprises commerce teams. When a sale completes through a protocol like UCP or ACP, there is no visit, no session, and no event on your site; revenue simply appears with no traffic behind it. The Google UCP guide covers that mechanic in detail. The practical consequence is that agentic revenue must be instrumented at the order layer or it does not get counted at all.

Layer one: reading crawler traffic in your logs

Server logs are the only reliable source here. If you are on a CDN, use its logs (Cloudflare, Fastly, CloudFront) since they see requests your origin may never receive from cache. The basic move is to grep access logs for known AI user agents and count them:

# Count AI crawler hits by bot, most active first
grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Google-Extended|Bytespider|CCBot|Amazonbot|Applebot|meta-externalagent|DuckAssistBot|MistralAI-User' access.log \
  | sort | uniq -c | sort -rn

# Which pages one specific bot fetches most
grep -i 'GPTBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Status codes served to AI crawlers (looking for 403s, 404s, and 5xx)
grep -iE 'GPTBot|ClaudeBot|PerplexityBot' access.log | awk '{print $9}' | sort | uniq -c

That third query is the one worth running first, because it is where problems hide. Crawlers receiving 403s are being blocked by a firewall or bot rule you may not know exists; crawlers receiving 404s are following stale links; crawlers receiving 200s but on pages that render empty without JavaScript are getting nothing useful. Any of those quietly removes you from AI answers while looking fine to human visitors.

Which bots are which (and why the distinction matters)

Every major provider now runs several crawlers with different jobs, and conflating them leads to genuinely bad decisions. Blocking a training crawler is a defensible content-rights choice. Blocking a search or user-triggered crawler removes you from AI citations and answers, which is usually the opposite of what a business wants.

For scale context: Cloudflare reported AI crawlers generating more than 50 billion requests per day across its network as of March 2025, just under 1% of all requests it saw. By January 2026 its comparison of crawl breadth found Googlebot still reaching far more unique URLs than any AI crawler, roughly 1.76 times more than GPTBot and 167 times more than PerplexityBot, so AI crawl volume is meaningful but nowhere near search-scale yet. Growth is the real signal: ClaudeBot approximately doubled its crawl rate between Q3 2025 and Q1 2026 on monitored sites, and AI search visits overall grew about 42.8% year over year, from 15.6 billion to 27.4 billion between Q1 2025 and Q1 2026.

Layer two: AI referral traffic

This layer does show up in analytics, because it is a human in a browser. Segment referrers for chatgpt.com, perplexity.ai, claude.ai, gemini.google.com, and copilot.microsoft.com, and treat the group as its own acquisition channel. Concentration is heavy: roughly 87% of AI referral traffic originates from ChatGPT, so a single-source segment gets you most of the picture, though Gemini referrals grew several hundred percent through late 2025 and deserve their own line.

// Classify AI referrers into a single channel dimension
const AI_SOURCES = {
  'chatgpt.com': 'ChatGPT',
  'chat.openai.com': 'ChatGPT',
  'perplexity.ai': 'Perplexity',
  'claude.ai': 'Claude',
  'gemini.google.com': 'Gemini',
  'copilot.microsoft.com': 'Copilot',
}

function aiChannel(referrer) {
  if (!referrer) return null
  try {
    const host = new URL(referrer).hostname.replace(/^www\./, '')
    return AI_SOURCES[host] || null
  } catch {
    return null
  }
}

Two honest caveats. First, this under-counts: someone who reads an AI recommendation and then types your name into a browser arrives as direct traffic, and no attribution model recovers that. Second, judge this segment on quality rather than volume, because AI referrals arrive pre-recommended and Adobe Analytics measured them converting roughly 42% better than traditional search traffic. A small, high-converting segment growing steadily is the shape to expect.

Layer three: agent transactions

Instrument this in your commerce backend, not your analytics. Practical approach: stamp every order with its origin at creation time, so agent-initiated checkouts carry a durable flag through to your reporting. If you expose an API or agent endpoints, require or encourage a client identifier on requests and record it with the order. Protocol-originated orders (UCP, ACP) should be tagged distinctly from your own API traffic, since they come from different surfaces and deserve separate lines in reporting.

The reason to do this before the volume justifies it is simple: you cannot reconstruct attribution retroactively. Teams that wait until agent revenue is obviously material then spend a quarter unable to say where any of it came from or which surfaces are working.

What to do with the numbers

Measurement is only worth the setup if it changes decisions. The four that usually follow from a first honest look:

1. Fix access errors first. Any 403s or 404s served to legitimate AI crawlers are pure lost visibility, and they are usually a CDN bot rule nobody remembered enabling.
2. Compare crawl coverage against your sitemap. Pages crawlers never fetch cannot be cited; if your most valuable pages are missing from crawler logs, that is an internal linking or discoverability problem.
3. Watch crawl frequency on updated pages. Answer engines weight freshness heavily, so a page you refreshed that has not been re-crawled will keep getting cited with its old content.
4. Correlate the layers. Rising crawler activity with flat referrals suggests you are being read but not recommended, which is a content and structure problem. Rising referrals with flat agent transactions suggests you are recommended but not transactable, which is a callable-actions problem.

That last point is the reason to instrument all three layers rather than whichever one is easiest. Each pair of adjacent layers has a specific failure mode between them, and the diagnosis only appears when you can see both sides. A business that measures nothing here is not neutral about the channel; it has decided, by default, that it will find out how big it got only in hindsight.