How to measure AI crawlers, referrals and transactions

AI-related activity does not appear in one clean analytics channel. Combine request logs, identifiable referrals and order records, while keeping crawls, customer visits and completed actions separate.

Reviewed October 1, 2026 · 7 min read

Many automated fetches never execute your analytics script, and analytics systems may filter bot activity. Browser agents can behave differently and may run scripts. Use CDN and server logs to inspect requests rather than interpreting an empty analytics segment as proof that no agents visited.

Measure three things separately: automated requests, identifiable visits from assistant links and transactions created through a known integration. They answer different questions and cannot be added together as if each represented a customer.

This guide shows where to collect each signal, how to classify it and how to avoid drawing conclusions the data cannot support.

The three layers, and what sees each one#

An API-driven order may have no website session, while a browser-assisted purchase may have one. Record origin in the backend where it is known, then reconcile it with analytics rather than assuming every protocol transaction is invisible to browser measurement.

Layer one: reading crawler traffic in your logs#

Use edge logs when a CDN serves or blocks requests before they reach the origin. The commands below are a quick diagnostic for a traditional combined access-log format. They classify claimed user-agents only; adapt the field parsing to your actual log schema.

# Claimed bot labels in a combined access log, not verified identities
rg -oi 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User' access.log | sort | uniq -c | sort -rn

# Requested paths for one claimed caller (combined-log field layout)
rg -i 'GPTBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Response codes for selected claimed callers
rg -i 'GPTBot|ClaudeBot|PerplexityBot' access.log | awk '{print $9}' | sort | uniq -c

Investigate repeated failures on important pages. A 403 may be an intended block, a 404 may be a bad URL, and a 200 may contain a challenge rather than useful content. Check the response body, rule and caller identity before diagnosing lost visibility.

Which bots are which (and why the distinction matters)#

Search, training and user-triggered fetching have different purposes. Keep them separate in reports so a training crawl does not get counted as a purchase-intent visit. The crawler policy guide links the provider documentation.

Start with your own baseline: requests by verified caller, important URLs reached, response status and time period. Report whether the logs are complete, sampled or retained for only a short window.

Do not compare unlike denominators. Total requests, unique pages, referred sessions and orders measure different activity. A rise in one can reflect a crawler configuration change rather than growing customer demand.

Keep the baseline long enough to see normal variation. Annotate deployments, access-rule changes and campaigns so you can investigate a shift without automatically crediting the latest content edit.

Layer two: AI referral traffic#

Segment identifiable assistant referrers, including the domains relevant to your audience. The example below uses exact host matching for a small list. It will not identify direct visits, stripped referrers or every assistant product.

const AI_SOURCES = {
  'chatgpt.com': 'ChatGPT',
  'chat.openai.com': 'ChatGPT',
  'perplexity.ai': 'Perplexity',
  'claude.ai': 'Claude',
  'gemini.google.com': 'Gemini',
  'copilot.microsoft.com': 'Copilot',
}

function aiChannel(referrer) {
  if (!referrer) return null
  try {
    const host = new URL(referrer).hostname.replace(/^www\./, '')
    return Object.hasOwn(AI_SOURCES, host) ? AI_SOURCES[host] : null
  } catch {
    return null
  }
}

Treat this as identifiable referral traffic, not a complete AI-influence total. Someone may read a recommendation and later arrive directly. Compare conversion and enquiry quality with suitable baselines, and disclose small samples rather than applying an industry average to your own business.

Layer three: agent transactions#

Record a known integration origin when creating an order or booking. Distinguish the protocol, authenticated client and entry channel where those facts are available. Do not trust a client-supplied 'AI order' flag as verified attribution, and do not store unnecessary personal or payment data just for reporting.

Use stable order identifiers to reconcile completed actions, cancellations and refunds without double counting retries. Some past attribution can be reconstructed from retained logs; information you never recorded may remain unknown. Label that uncertainty explicitly.

Use platform reports for the questions they answer#

Search and merchant platforms may provide their own visibility or AI-related reports. Check the definition, account availability, geography and reporting window of the feature you actually have. Do not assume every account can isolate every AI placement.

A platform's share-of-voice metric, your request count and your referral sessions have different denominators. Keep them side by side with their definitions rather than forcing them into a single funnel whose steps cannot be matched.

What to do with the numbers#

Use the evidence to choose an investigation, not to jump straight to a cause:

1. Access failures: identify the caller, path and matched rule, then decide whether the result is intended.
2. Coverage gaps: compare requested pages with important URLs, while accounting for log sampling, caches and alternate data sources.
3. Stale information: inspect the current page, feed and cited source before assuming a recrawl delay.
4. Low referrals despite requests: consider crawl purpose, answers without clicks and missing attribution, not just content quality.
5. Weak conversions: inspect the landing page and customer journey; low agent-order counts alone do not prove an API is needed.

The useful report distinguishes what happened from what you infer. Show known requests, identifiable visits and verified outcomes, then list the gaps that prevent stronger conclusions. That makes the next investment easier to assess.