Should you block AI crawlers? Allow, block or cite
Should you block AI crawlers on a client's site? Training bots, search bots and user-triggered agents are different, and each deserves its own decision.
A client reads that AI companies are scraping the web to train their models, and asks you to block them. Another client, in the same week, asks why their business never shows up when someone asks an assistant for a recommendation. They're asking for opposite things, and both are reasonable.
The reason they can both be right is that "AI crawlers" isn't one thing. There are at least three kinds of bot, doing three different jobs, and each one deserves its own decision.
The three kinds of bot
Training crawlers. These collect content to train future models. The words go into a dataset, and the site gets nothing back directly: no visit, no citation. Examples include GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), meta-externalagent (Meta) and Bytespider (ByteDance). Google-Extended and Applebot-Extended are slightly different: they're robots.txt tokens that control whether content Google or Apple already crawls may be used for training, rather than separate bots.
Search and answer bots. These fetch pages so an assistant can answer questions and cite sources. OAI-SearchBot, Claude-SearchBot and PerplexityBot are examples. When your client's page is cited in an answer, it's usually one of these that fetched it.
User-triggered agents. These fetch a page because a person asked their assistant to, right now. "Read this page and summarise it." "Check whether this restaurant is open tonight." ChatGPT-User, Claude-User and Perplexity-User are examples. They behave much more like a person with a browser than like a crawler. OpenAI and Perplexity say robots.txt may not apply to their user-triggered fetchers, because a person asked for the page. Anthropic says its bots, Claude-User included, honour it.
Bot names change, and operators add new ones. Check each operator's own documentation on the day you write a policy, rather than copying a list from a blog post, including this one.
The trade for each
Blocking training crawlers protects the content from being used to train models. For a publisher, a writer or anyone whose words are the product, that matters. For a local plumber it matters much less. The cost is small for most sites: training rarely brings visitors directly.
Blocking search bots costs citations. If an assistant can't fetch the page, it can't quote it or link to it, and it'll answer the question using someone else's site. For a business that wants to be found, that's a real loss.
Blocking user-triggered agents breaks someone's request. A person asked their assistant to look at this site, and the site said no. That's closer to refusing a visitor at the door than to opting out of a dataset. It's rarely what a client actually wants.
robots.txt is a request
robots.txt is a text file that asks bots to stay away from certain things. Well-behaved bots read it and comply. Nothing forces them to.
Enforcement is a different thing. A site, or the network in front of it, can look at the user agent on each request and refuse the ones it has decided to block, with an error instead of the page. That's edge blocking. It doesn't stop a bot that lies about who it is, but it turns a polite request into a rule for every bot that identifies itself honestly, which the major operators say theirs do.
If a client really wants training crawlers kept out, ask whether their policy is a request or is enforced. Most sites only have the request.
Content-Signal, briefly
Content-Signal is a newer line that goes inside robots.txt. Rather than saying which bots may fetch what, it states what the content may be used for: search, as input to AI answers, or for training. Each can be yes or no. It's a statement of terms more than a technical control, and it isn't yet clear which operators act on it. It's cheap to add and puts the client's position on record.
Our defaults, and why
On every site hosted on Contentive, the defaults follow the trade above. Search and answer bots are allowed, and so are user-triggered agents, so the site can be cited and people's assistants can read it. Training crawlers are blocked by default, and so is Bytespider. The block isn't just a line in robots.txt: a blocked crawler that identifies itself gets a 403 at the edge. A Content-Signal line says search yes, AI input yes, training no.
We chose those defaults because they match what most agency clients want without having to think about it: be found, don't be trained on. Clients who want something different can have it. Per-bot allow and block overrides are a Studio setting with the SEO app installed. More on what else a site needs in what an AI-ready website actually means.
The client conversation
This is the client's decision, not yours. It's about their content and their business, and the right answer depends on how they make money.
Put it in front of them as a short choice, not a technical briefing. Three questions cover it. Do you want assistants to cite and link to your site? Do you mind your content being used to train AI models? Is there anything on the site you'd rather wasn't summarised elsewhere? Write down the answers, set the policy to match and add it to the handover. It belongs on the pre-launch checklist too, so it's decided before launch rather than after an article worries them.
What it costs
There's no free option. Allowing everything gives away the content. Blocking everything makes the site invisible to a growing way people find things. The middle path, which we default to, still lets assistants quote the client's words without sending a visit. Some clients will dislike that, and they're entitled to.
A one-line policy by client type
Publisher or writer: block training, allow search with care, allow user-triggered agents.
Local business: allow search and user-triggered agents, block training. Being cited is the point.
SaaS or product company: allow search and user-triggered agents, and decide on training deliberately, because some want their docs in the models.