Skip to content

You changed your AI crawler or opt-out settings. How do you measure what it did to AI visibility?

Google will read one shared AI training opt-out from 2027. How to test any crawler or opt-out change: a baseline, a change log and readings by engine.

By Published 14 min read
Small crawler robots queue at a gate in front of a web page while a figure reads a chart split into before and after.

The short answer

To measure what an AI crawler or opt-out change did to your AI visibility, take a four-week baseline first, change one setting at a time and log the date. Then read crawler hits in your server logs, Search Console and Bing reports, and the same prompt set on each engine, and compare against unchanged control prompts.

Key takeaways

  • Google said on 30 October 2026 that it aims to support a shared AI training opt-out in robots.txt in 2027, and will treat that opt-out as a Google-Extended disallow when a site says nothing else about Google-Extended.
  • A training opt-out, a search crawler block and Google's AI features opt-out act on different systems, so each one needs its own measure and its own waiting time.
  • Crawlers for AI training made 52% of crawler requests on Cloudflare's network in June 2026, up from 22% in spring 2025, so server logs are often the first place a robots.txt change shows.
  • Ahrefs found that 14.4% of 66.6 million free, logged-out ChatGPT responses in August and September 2026 triggered a web search, so a crawler change can only reach answers that search.
  • Our view: change one setting at a time, keep a dated change log and judge the result against control prompts, because model updates and answer variance move the numbers too.

What did Google announce on 30 October?

Google announced that it will read one shared AI training preference in robots.txt instead of asking sites to name each AI company separately. The post on the Google Search Central blog, dated 30 October 2026, says the control is a shared "policy token" based on work from the AIPREF working group at the Internet Engineering Task Force.1 Google says it aims to support it, along with other AI providers, in 2027.1

Two details matter for anyone who measures AI visibility. If a site opts out of training with the shared token and does not mention Google-Extended, Google will treat that as a Google-Extended disallow.1 Company-specific controls, Google-Extended included, keep working alongside it. The post also notes that sites which specifically allow Google-Extended are eligible to appear in Gemini Apps responses through grounding.1

So one line in robots.txt will soon speak to several AI providers at once. That is convenient, and it makes the effect of a change harder to read, because one edit can touch several engines in different ways.

Nothing changes for your site today. The control is planned, not live, and the draft it rests on is still being discussed. The useful window is the months before it arrives, when you can measure where you stand without a new rule muddying the readings.

Prior work, and what this guide adds

Others have tested opt-outs before. Search Engine Roundtable reported on 30 October 2026 on a small case study by Cyrus Shepard, who used Google's AI features opt-out on one site and compared impressions and clicks before, during and after.2 Our earlier notes cover whether to block AI training crawlers and what Search Console's AI reports and opt-out do. This guide adds the test design: how to read any crawler or opt-out change across all five engines without fooling yourself.

A disclosure first. Sigzen AI sells AI visibility audits at published prices, and some studies cited here come from companies that sell tools in this category.

Which setting controls which surface

Each AI setting acts on a different system, so start by writing down what your change can and cannot reach. A block on a training crawler changes what future models learn. A block on a search crawler changes what engines can fetch for live answers. Google's AI features opt-out changes what Google shows. The measure follows from which of these you touched.

OpenAI documents separate crawlers for separate jobs: GPTBot for training, OAI-SearchBot for search results and ChatGPT-User for fetches a person asks for, each with its own robots.txt rule.3 Google works differently. Its guidance says Googlebot controls crawling for Search, AI features included, and names Google-Extended as the control for training and grounding in some of its other systems.4

Setting you changeWhat it can affectWhere you see it first
Block a training crawler (GPTBot, Google-Extended)What future models learn from your pagesServer logs; answers only after a new model
Shared training token (from 2027)Training for every provider that honours itServer logs; Gemini Apps answers for Google
Block a search crawler (OAI-SearchBot)Whether live answers can fetch your pagesServer logs; cited links in searched answers
Google's AI features opt-outYour pages in AI Overviews, AI Mode, DiscoverSearch Console AI impressions
CDN bot rules (Search, Agent, Training)Any crawler the CDN classifies and blocksCDN analytics; server logs

Sources: OpenAI's crawler documentation3, Google's AI features guidance4, Google's 30 October post1. The last column is our reading.

The CDN row catches people out. Cloudflare's July 2026 changelog splits AI traffic into Search, Agent and Training controls.5 From 15 September 2026, new domains on Cloudflare block Training and Agent traffic by default on pages that display ads, while Search stays allowed.5 A site can change its AI exposure without anyone editing robots.txt.

Our view: before testing anything, list every place a bot rule lives, including robots.txt, the CDN, the firewall and the Search Console setting. A test that ignores one of them measures the wrong change.

Why is a before-and-after reading hard to trust?

A before-and-after reading is hard to trust because AI visibility moves for many reasons at once, and a settings change is rarely the largest. Engines update models, change how often they search and vary their answers from run to run. A drop in the week after your change may have nothing to do with it.

Start with how much an answer depends on live fetching at all. Ahrefs, which sells SEO and AI visibility tools, tracked 66.6 million free, logged-out ChatGPT responses in August and September 2026 and found that 14.4% triggered a web search.6 In that sample, most answers came from what the model already held, and a crawler change cannot reach those until a new model is trained.

Models also change under you. OpenAI's ChatGPT release notes record GPT-6 Sol becoming the model for paid tiers from 7 October 2026, with GPT-6 Luna reaching Free and Go users from 8 October 2026.7 A robots.txt edit made that week would have been read against a different model on each side of the change.

What moves an AI visibility number besides your change

  1. Model update
  2. Search rate shifts
  3. Run-to-run variance
  4. Rivals publish
  5. Your settings change
  6. Reading you see
Illustrative: the change under test is highlighted; the other stages move the same number.

Answer variance is the third trap. MaxAEO, which sells AI visibility measurement, argues that repeat runs of one prompt are clustered: they resemble each other, so extra repeats add less information than new prompts would.8 A small prompt set run many times can look precise and still swing on a handful of prompts. Our note on why one ChatGPT answer can differ from a report covers the same effect.

The test design, step by step

A sound test of a settings change has five parts: a fixed prompt set, a baseline, a dated change log, one change at a time and a control. None of these needs special tools. Each one removes a reason to doubt the result.

Testing one crawler or opt-out change

  1. Freeze the prompt set

    The buyer questions you track, unchanged for the whole test, plus control prompts the change cannot affect.

  2. Take a baseline

    Four weeks of readings on all five engines, several runs per prompt, before you touch anything.

  3. Log the change

    Date, time, exact rule, where it lives and who made it. Note any model update the same week.

  4. Change one thing

    One crawler, one rule or one setting. Wait before the next change.

  5. Read by surface

    Crawler hits, Google and Bing reports, prompt runs and referrals, each on its own timetable.

Illustrative: a test design, not a measured result.

The control prompts do the most work. Pick questions where your pages are rarely the source, such as category definitions answered from encyclopaedias, or prompts about a sister brand on another domain. If those move as much as your tracked prompts, the engine changed, not your setting.

The change log should outlive the test. Six months from now, someone will ask why a number fell in November, and a dated line saying "blocked GPTBot on all paths" answers it in seconds. Keep it next to the dashboard, not in someone's inbox.

Why one change at a time

Changing several settings in one week makes the result unreadable. If you block a training crawler, tighten a CDN rule and switch on Google's opt-out together, a fall in citations could come from any of them. Space changes at least two weeks apart for search crawlers, and keep a training change apart from any search change.

Our view: when a legal or commercial deadline forces several changes at once, still log them separately and take the baseline anyway. You lose the ability to attribute the effect, but you keep the ability to show its size.

What should you measure on each surface?

You should measure each surface with the source closest to it: server logs for crawlers, platform reports for Google and Bing, prompt runs for what answers say, and analytics for visits. No single report covers all five engines, and each one counts something different. Read them side by side, never added together.

Server logs show the change first. On Cloudflare's network, crawlers for AI training made 52% of crawler requests in June 2026, up from 22% in spring 2025.9 On many sites that means AI crawlers are a visible share of log lines, so a robots.txt rule that works shows up within days as fewer requests from the blocked agent. Check that blocked agents stop, and that the ones you allowed keep coming.

Verify who is asking, too. A user agent string is easy to copy, and some scrapers borrow the name of a well-known crawler. Where the operator publishes the addresses its crawler uses, match your log lines against them before you count a request as that crawler. Otherwise a block can look broken when the real crawler has already stopped.

Signals around a settings change

Where the numbers come from

52%of crawler requests on Cloudflare's network came from AI training crawlers, Jun 2026Cloudflare, spring 2025–Jun 2026
14.4%of free, logged-out ChatGPT responses triggered a web searchAhrefs, Aug–Sep 2026
68.01%of US Google searches ended without a clickSparkToro, Jan–Apr 2026
Sources: Cloudflare9, Ahrefs6, SparkToro10.

Google's side reports impressions only. Search Console's AI reports show impressions in AI Overviews, AI Mode and Discover, not clicks.11 Search Engine Roundtable reported on 2 November 2026 that Google's Gary Illyes said the team wants to give more data but has nothing to announce, which the site read as a sign that click data is not imminent.12 Plan on impressions for now.

Microsoft reports citations. Bing Webmaster Tools added an AI Performance report in public preview in February 2026, covering Microsoft Copilot, AI-generated summaries in Bing and select partner integrations.13 ChatGPT, Gemini and AI Mode answers themselves have no owner's report, which is where your own prompt runs come in.

Two ways to read a settings change

One headline number

  • Total AI referrals, week on week
  • No baseline, no control prompts
  • Model updates read as your effect
  • Several changes blended together

One reading per surface

  • Crawler hits by agent from logs
  • Search Console and Bing reports
  • Prompt runs by engine with controls
  • Each read on its own timetable
Illustrative: our recommended approach is highlighted.

Prompt runs: what to record

For each prompt and engine, record whether your brand was named, whether your pages were cited, and whether the answer described you correctly. Keep cited URLs, because a search-crawler block should remove your own pages from citations while mentions sourced from other sites may stay. Our guide to picking a rival set explains why you should record competitors in the same runs.

Visits still matter, but they are a weak test. SparkToro, using Similarweb's US clickstream panel, found 68.01% of US Google searches ended without a click in January to April 2026.10 When most searches end without a click, a change in AI exposure can move what buyers read about you long before it moves sessions.

Reading the result without fooling yourself

Read the result as a difference between your tracked prompts and your control prompts, by engine, over the same weeks. If tracked prompts fall and controls hold, the change probably did it. If both fall, an engine moved, and your change may have done nothing at all.

Report a range, not one figure. With a few dozen prompts, the difference between two four-week windows will carry real uncertainty, and a single percentage hides it. Our note on estimating AI Overview traffic loss uses the same approach for click-through data.

Two panels show a switch in its off and on positions with a different number of answer cards beneath each one.
A control stream that the change cannot touch is what lets you read the one it can.

Watch for effects outside the AI surfaces. A Google AI features opt-out removes your links from AI Overviews and AI Mode, but the case study reported on 30 October also looked at regular organic clicks during the test.2 Include your ordinary search figures in the reading, not only the AI reports.

Keep cited and uncited effects apart, too. In Seer Interactive's data for 53 brands from January 2025 to February 2026, pages cited in an AI Overview earned about 120% more organic clicks per impression than when they were not cited.14 That is a correlation, but it suggests that losing a citation can cost more than the AI impression itself.

Decide in advance what result would make you undo the change. For a search crawler block, that might be your own pages leaving the citations for prompts that bring buyers. Writing the threshold down before the reading stops a disappointing number from being explained away afterwards.

Our view: a result you cannot reverse is not a test. Where you can, keep the change reversible for the first month, so a bad reading can be undone while the baseline is still fresh.

How long before a change shows up?

How long a change takes depends on which system it touches. Crawler hits change within days of a robots.txt or CDN edit. Searched answers follow within weeks, once engines re-fetch. Training changes reach answers only when a new model ships, which can take months and arrives on the provider's schedule, not yours.

When to read each signal after a change

  1. Server and CDN logs: blocked agents stop, allowed agents continue.

  2. Searched answers: cited URLs change on engines that fetch live.

  3. Search Console AI impressions and Bing AI Performance citations.

  4. Full prompt-set reading against the baseline and the controls.

  5. Answers from memory: the only place a training change can show.

Illustrative: a schedule based on how each system works, not a measured result.

That makes a training opt-out almost impossible to read in the short term. Your logs will confirm the crawler stopped, and your answers will look the same for months. That is expected, and it is not evidence that the opt-out failed or that it was harmless.

Gemini is the exception to watch. Because Google links Google-Extended to grounding in Gemini Apps, a training opt-out that maps to a Google-Extended disallow could change what Gemini Apps cite from your site.1 Read Gemini separately from Google's AI Overviews and AI Mode, which Googlebot governs.4

What to do this month

This month, set up the measurement before the 2027 control arrives, so you have a baseline when your team decides how to use it. The decision itself can wait. A missing baseline cannot be recovered later.

  1. List every place a bot rule lives: robots.txt, CDN, firewall and the Search Console setting.
  2. Write down the current rule for each AI crawler and the date it was last changed.
  3. Freeze a prompt set with control prompts, and start four weeks of baseline runs on all five engines.
  4. Set up a log view of crawler requests by user agent, so you can see a rule take effect.
  5. Agree that one setting changes at a time, with a dated entry in the change log.
  6. Book the eight-week reading before the first change goes live.

Our view: most companies will touch their AI crawler rules at least once in 2027, and many will do it under time pressure. A baseline taken now costs little and turns that later decision from a guess into a measured trade.

Sources

  1. Google Search Central blog, a shared AI training control in robots.txt (30 Oct 2026)
  2. Search Engine Roundtable, report of an AI features opt-out case study (30 Oct 2026)
  3. OpenAI, Overview of OpenAI crawlers (developer documentation)
  4. Google Search Central, AI features and your website (updated Dec 2025)
  5. Cloudflare changelog, New options to manage AI traffic: Search, Agent and Training controls; new-domain defaults from 15 Sep 2026 (Jul 2026)
  6. Ahrefs, ChatGPT query fan-out study: 66.6M free, logged-out GPT-5.6 Luna responses, Aug–Sep 2026, all countries; factual subset 20,000 US prompts; citation ranks US, Mar–Sep 2026 (Oct 2026)
  7. OpenAI, ChatGPT release notes: GPT-6 Sol for paid tiers from 7 Oct 2026 and GPT-6 Luna for Free and Go from 8 Oct 2026 (Oct 2026)
  8. MaxAEO, AI visibility sample size and clustered runs (Jul 2026)
  9. Cloudflare, crawler mix on its network, spring 2025–Jun 2026 (Jul 2026)
  10. SparkToro, less than one-third of Google searches send a click: Similarweb US desktop and mobile clickstream, Jan–Apr 2026 (Jun 2026)
  11. Search Engine Journal, Google Search Console AI reports rolled out worldwide (31 Aug 2026)
  12. Search Engine Roundtable, Google comment on click data in Search Console AI reports (2 Nov 2026)
  13. Microsoft Bing, AI Performance in Bing Webmaster Tools, public preview (Feb 2026)
  14. Seer Interactive, AI Overviews and CTR, 2026 update: 53 brands, Jan 2025–Feb 2026 (Apr 2026)

Questions readers ask

  • Google's shared AI training opt-out is unlikely to remove a site from AI Overviews or AI Mode, because Google says it will treat that opt-out as a Google-Extended disallow, and its guidance places AI features in Search under Googlebot controls. The separate Search Console setting is the control for leaving those features. Gemini Apps grounding is the place a training opt-out could show.

Keep reading

Free. Enter your domain and Sigzen AI checks how five engines answer your buyers' questions, against up to three competitors.

No account needed. Two minutes to submit; we email the full five-engine report when the check has run.