Access

Does the site tell agents the rules — robots.txt, per-bot rules, Content-Signal, ai.txt, and TDMRep.

• site-wide file

  robots.txt

  A well-formed robots.txt: directives inside User-agent groups, known keys, proper key: value syntax.

  ✓ correct ✗ broken
• site-wide file

  Per-bot AI rules

  Correctly-spelled, non-contradictory groups for named AI bots (GPTBot, ClaudeBot, …) in robots.txt.

  ✓ correct ✗ broken
• site-wide file

  Content-Signal

  Cloudflare/AIPREF Content-Signal directives in robots.txt with valid yes/no values inside a group.

  ✓ correct ✗ broken
• site-wide file

  ai.txt

  A Geodocs ai.txt with parseable Key: value directives, absolute Source-url, valid attribution flags.

  ✓ correct ✗ broken
• site-wide file

  TDMRep

  A W3C TDMRep /.well-known/tdmrep.json: an array of rules with location and a numeric tdm-reservation.

  ✓ correct ✗ broken