robots.txt
A well-formed robots.txt: directives inside User-agent groups, known keys, proper key: value syntax. (site-wide file)
Broken variant
A rule before any group, a bare User-agent:, a missing colon, and an unknown directive.
Defect metrics
robotsTxtDirectiveBeforeGroup |
robotsTxtMissingColon |
robotsTxtUnknownDirective |
robotsTxtEmptyUserAgent |
robotsTxtOversized |