robots.txt broken variant
A well-formed robots.txt: directives inside User-agent groups, known keys, proper key: value syntax.
Site-wide file — one per host.
This signal isn't a page; the container fetches it at a fixed path, and the
correct/broken/authbroken variant is chosen by the request host. On this host the canonical path(s)
below serve the broken variant. Add
?mode=… to override locally.
Canonical path(s)
Broken variant
A rule before any group, a bare User-agent:, a missing colon, and an unknown directive.
Defect metrics
| Defect boolean |
|---|
robotsTxtDirectiveBeforeGroup |
robotsTxtMissingColon |
robotsTxtUnknownDirective |
robotsTxtEmptyUserAgent |
robotsTxtOversized |
Fetch the live file: https://agentic-authbroken.crawler-test.com/robots.txt