The problem this solves
Plenty of sites investing heavily in citable content are invisible for a reason that has nothing to do with quality: the AI crawlers cannot get in. A robots.txt written years ago blocks everything unfamiliar, a CDN bot-protection rule silently challenges GPTBot, key content renders only through JavaScript the crawlers never execute, and llms.txt simply does not exist. None of this shows up in normal SEO tooling, because the classic search crawlers get whitelisted long ago.
Access is the zeroth requirement of citation share: an engine cannot cite a page it never fetched. The audit is mechanical and the fixes are mostly small text changes, which is exactly why they get skipped - nobody owns them, marketing assumes engineering handles crawlers, and engineering assumes the SEO tool would have said something. A mission can check every gate in one pass, attach the evidence per crawler and per page, and put the corrections where engineering actually reviews work: a pull request with reasoning on every line.
How the mission runs
- Enumerate the gatekeepers. The mission checks robots.txt and llms.txt against the current roster of AI crawlers - GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest - reading what each is allowed, disallowed, or left ambiguous.
- Test real fetches. Key pages are fetched the way crawlers see them, surfacing CDN and firewall challenges, redirect loops, and soft blocks that a browser never shows you. What a crawler receives is recorded, page by page.
- Check parseability. Reachable is only half: the audit inspects whether the substance of each page survives without JavaScript execution - the failure mode where a crawler fetches your page and finds an empty shell.
- Open the PR. Fixes land as a pull request on a new branch: robots.txt allowances, an llms.txt that maps your citable content, and notes on anything needing infrastructure changes beyond a file edit. Every change carries its reasoning for review.
The prompt
This is the exact objective the agent receives. Swap the obvious placeholders for your own domain, segment or channel and run it as-is from the console, Slack, or the API.
What comes back
A pull request with the concrete access fixes - robots.txt rules, an llms.txt built from your actual content map - plus an audit report showing, per crawler and per key page, what was reachable, what was blocked, and what rendered empty. Merge it and the zeroth requirement of AI visibility is settled.
Make it yours
- Point it at a staging domain first if your robots policy is deliberately restrictive and needs a debate before changing.
- Add a quarterly re-run as a standing mission: crawler rosters and CDN rules both change under you.
- Extend the parseability check to your docs subdomain, where JS-only rendering is most common and citations matter most.
Frequently asked questions
Is opening access to AI crawlers always the right call?
It is a policy choice, and the audit treats it as one: the report shows what each rule currently does so the decision is explicit. If you choose to keep some crawlers out, the PR encodes exactly that intent instead of an accident of old configuration.
What is llms.txt and does it matter?
A plain-text map of your most citable content, placed at the site root for AI systems to read. Adoption among engines varies, but it costs one file, it concentrates crawler attention on the pages you want cited, and this mission generates it from your real content rather than a template.
Will the PR touch anything risky?
The changes are text files by design, opened on a new branch and never merged by the agent. Anything that needs infrastructure work - a CDN rule, a rendering change - is written up in the report with its evidence instead of attempted.