Can AI crawlers read your site?

Questions about the crawler check.

• Which AI crawlers does the free crawler check look for?

  The free crawler check looks for sixteen named crawlers: the OpenAI, Google and Microsoft bots that feed the five engines Sigzen AI measures, and the crawlers of Perplexity, Anthropic, Apple, Amazon, Meta, ByteDance and Common Crawl. The list carries a version date and is reviewed against the vendors' published names each quarter.

• Does blocking GPTBot remove a site from ChatGPT answers?

  Blocking GPTBot stops OpenAI reading the site for training, not for answers: ChatGPT's search answers are fetched by OAI-SearchBot and ChatGPT-User, which obey their own robots rules. A site that blocks all three can still be named from what other sites say about it, but never from its own pages.

• What does it mean when a site has no robots.txt?

  A site with no robots.txt allows every crawler by default, because the file only ever adds restrictions. For AI visibility that is the open setting, so nothing needs fixing. A robots.txt is worth adding only to keep crawlers out of admin, search or API paths, and it must then allow the rest.

• Should a site block AI training crawlers but allow AI search crawlers?

  A site can block training crawlers and allow search crawlers, because the vendors name them separately: GPTBot and ClaudeBot train models, while OAI-SearchBot, Claude-SearchBot and Googlebot fetch pages for answers. Sigzen AI's advice is to decide the two separately; blocking the search crawlers removes you from the answers you want.

• Does the crawler check read anything besides robots.txt?

  The crawler check reads only robots.txt, over https and then http, with a six-second timeout and a size cap, and stores nothing about the domain. It does not crawl pages, read meta tags or test server-side blocking, so a crawler the file allows can still be refused by a firewall; Sigzen AI's audit checks that from the engines' answers.