What an AI-ready website actually means
What an AI-ready website actually means: a crawler policy, a sitemap, an llms.txt and clean structured pages. What matters, what's hype, and what we serve.
A client forwards an article that says their website needs to be "ready for AI", and asks what you're doing about it. The article doesn't say what ready means. It mostly says that everything is changing.
Here's the plain version. Most of what makes an AI-ready website is the same thing that made a site good for search engines ten years ago: clear structure, clear policy and pages that load as real HTML. A few newer pieces sit on top. Some of them are proven, some are reasonable bets and some are hype. It's worth knowing which is which before you sell any of it.
What's proven and what's speculative
Proven: answer engines and AI assistants fetch pages, read them and cite them. The major operators say their crawlers honour robots.txt, though some say a fetch a person asked for may not follow it. They do better with pages whose content is in the HTML rather than assembled by JavaScript after load. None of that is new.
Reasonable bets: telling different bots different things, and stating your terms for how content may be used. Neither is guaranteed to be honoured by every operator, but both cost little, and both put the client's choice on record in a place machines are expected to read.
Speculative: llms.txt. It's a proposed file that gives a language model a short, curated map of the site. It's cheap to produce and does no harm, but adoption by the major model providers is unproven. We haven't seen public evidence that the major assistants read it. Anyone who sells it as a ranking factor is guessing.
The parts of an AI-ready website
A crawler policy. Not one rule for every bot. Training crawlers, search and answer bots and user-triggered agents do different things, and they deserve different answers. Should you block AI crawlers? goes through the three kinds and the trade for each.
A current sitemap. A list of every public page, generated from the content rather than maintained by hand. A stale sitemap is worse than none, because it points crawlers at pages that no longer exist and leaves out the ones that do.
An llms.txt. Optional and speculative, as above. If you produce one, generate it from the site's own content so it can't fall out of date.
Structured content. Pages whose parts mean something: a title, a summary, an address, an opening time, a price, a list of services. When a site is built from a content model rather than freeform pages, a machine reading it can tell a phone number from a paragraph. What is structured content explains why that matters well beyond AI.
Fast, server-rendered HTML. If the words only appear after a heavy script runs, many crawlers never see them. Render on the server, keep pages light and the content is there on the first request.
Markdown views of pages
One idea that comes up often is serving a plain markdown version of every page, at an address like the page's own with .md on the end. An agent asking for the page gets clean text with headings and links, without navigation, scripts or styling to wade through.
It's a sensible idea. It saves an agent work and reduces the chance of it misreading a layout. It also depends on the content being structured in the first place, because a markdown view of a page built in a visual canvas is mostly noise. We don't serve markdown views today, so treat this as a concept to understand, not a feature to look for on our sites.
What every site we host serves today
Every site hosted on Contentive gets the following, generated from its published content on the site's own address. No plugin, no setup.
A
robots.txtwith an AI-crawler policy. Search and answer bots and user-triggered agents are allowed. Training crawlers, including GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent and CCBot, are blocked by default, and so is Bytespider. A blocked crawler that identifies itself gets refused at the edge, not just asked politely.A Content-Signal line. Inside
robots.txt, it states that the content may be used for search and as input to AI answers, but not for training.A
sitemap.xml. Built from published, routable pages, so it's always current.An
llms.txtand anllms-full.txt. Generated from published content. Member-only pages are left out of both.
Because the theme renders on the server from a content model, the pages themselves are structured HTML too. The per-bot policy can be changed in the Studio with the SEO app installed.
What it costs
The technical parts are cheap. The decision isn't.
An allow policy means your client's words get quoted elsewhere, often without a visit. An assistant may answer a question using their service page, and the person asking may never click through. For some businesses that's free reach. For a publisher that sells its words, it's a leak. Blocking means fewer citations and less visibility in AI answers.
There's a smaller cost too. Every extra file is something that has to stay accurate, and a hand-maintained llms.txt that describes last year's services is worse than none. Generate these files or don't publish them.
The policy itself is a business decision about the client's content, so it belongs with the client. Your job is to explain the trade and set the policy they choose, not to pick one on their behalf because a default happened to be on.
Check a client's robots.txt today
Pick one live client site. Open its robots.txt in a browser. Look for GPTBot, and for any mention of AI at all. If there isn't one, the site's policy is whatever each bot's operator decides it is. That's the conversation to have with the client this week.