The problem this solves
Most sites are running a crawler policy nobody ever wrote. The robots.txt was copied from a template years ago, nothing addresses AI crawlers at all, and the site has no llms.txt telling AI systems what it is or where its best content lives. The result is policy by accident: crawlers you would gladly admit hitting generic rules, crawlers you might want to limit roaming free, and the pages you most want machines to understand given no signposting whatsoever.
The stakes have shifted under the old file. AI systems now read sites to answer questions, cite sources, and describe what companies do, and a site's crawler policy has become part of its distribution strategy. Blocking everything means absence from answers people actually read; admitting everything without guidance means being represented by whatever a crawler happened to parse well. The sensible middle - deliberate admission with deliberate emphasis - requires decisions most teams have simply never sat down to make.
The reason they have not is that the domain is genuinely unfamiliar: which crawlers exist, what each one feeds, what llms.txt can express, and how directives interact. This playbook turns that research into a reviewable artifact: both files written for your specific site, every directive explained in plain language, delivered as a pull request your team can debate line by line. The policy becomes deliberate, and it merges only when a human agrees with it.
How the mission runs
- Assess how machines see the site today. AI Visibility establishes the baseline: how AI systems currently read, describe, and cite the site, which crawlers the existing robots.txt admits or blocks, and where the current accidental policy is helping or hurting. The recommendations that follow are grounded in this observed present.
- Draft the admission policy. The robots policy is written deliberately: which crawlers to admit, which to limit, and why each call was made, based on what each crawler feeds and what your site gains or risks by its access. Every directive gets a plain-language rationale, so the policy is legible to the whole team.
- Write the llms.txt. The llms.txt is authored for your actual site: what it is, who it serves, and where its highest-value content lives, structured so AI systems parse and use it. The highlighting choices - which pages deserve machine attention first - are stated as choices, ready for your team to confirm or amend.
- Open the PR with both files. Both files arrive as a pull request on GitHub, each directive annotated with its explanation, and the PR description laying out the policy's logic end to end. It is a small diff carrying a real decision, which is exactly what review exists for, and nothing merges itself.
- Hand the decision to the team. The PR waits for review: the annotations make each directive debatable on its merits, disagreements resolve as review comments, and the merged result is a crawler policy the team actually chose. A follow-up visibility check after merge can confirm the changed policy is being observed.
The prompt
This is the exact objective the agent receives. Swap the obvious placeholders for your own domain, segment or channel and run it as-is from the console, Slack, or the API.
What comes back
A pull request containing a written llms.txt and a deliberate robots policy for your site: every directive annotated with its rationale, the admission and emphasis choices laid out as choices in the PR description, and the whole policy grounded in an assessment of how machines read the site today. Your team reviews, debates, and merges on its own judgment, replacing years of accidental policy with a page of deliberate one.
Make it yours
- Re-run the assessment quarterly: new crawlers appear steadily, and each run proposes a small amendment PR when the roster or your priorities have shifted.
- Pair the merge with a follow-up AI-visibility check a few weeks later, measuring whether admission and signposting changed how answers describe and cite you.
- Extend the llms.txt emphasis section as part of every major content launch, so new flagship material gets machine signposting the day it ships.
Frequently asked questions
Why does this need a pull request at all?
Because crawler policy is a real decision with distribution consequences, and decisions deserve review. The PR format puts every directive in front of your team with its rationale attached, makes disagreement cheap to express, and guarantees the policy that ships is one a human explicitly approved. Nothing merges itself.
How do you decide which crawlers to admit?
From what each crawler feeds and what your site stands to gain or lose by its access, weighed against your goals: presence in AI answers, control over reuse, server load. The draft states each call's reasoning in plain language, so your team can override any individual judgment with full context.
Does llms.txt actually change anything?
It is an emerging convention: adoption among AI systems is real and uneven, which the recommendations treat honestly. The file costs a page of text, states your site's structure and priorities in machine-legible form, and positions you for the systems that honor it, while the robots policy does the enforceable work.