Per-bot AI rules broken variant
Correctly-spelled, non-contradictory groups for named AI bots (GPTBot, ClaudeBot, …) in robots.txt.
Site-wide file — one per host.
This signal isn't a page; the container fetches it at a fixed path, and the
correct/broken/authbroken variant is chosen by the request host. On this host the canonical path(s)
below serve the broken variant. Add
?mode=… to override locally.
Canonical path(s)
Broken variant
A misspelled token (GPT-Bot), a group that both Allows and Disallows /, an empty group, and a duplicated group.
Catalogued flags
| Flag | Kind | Severity |
|---|---|---|
perBotRuleMisspelledToken | defect | medium |
perBotRuleContradictoryAllowDisallow | defect | high |
perBotRuleEmptyGroup | informational — fails nothing | — |
perBotRuleDuplicateGroup | informational — fails nothing | — |
Fetch the live file: https://agentic-mcp-strict.crawler-test.com/robots.txt