Crawler evidence starts with executable fixtures.
Seven versioned robots-rule cases run through the same local evaluator as the public tool. Expected and actual decisions stay visible; no external crawler leaderboard is implied.
Executed result matrix
| Fixture | Rule job | Token and target | Expected | Actual |
|---|---|---|---|---|
REP-01 | Exact product group replaces wildcard | ExampleBot/public/report | ALLOW | ALLOW · PASS |
REP-02 | Longest matching rule wins | ExampleBot/private/preview/index.html | ALLOW | ALLOW · PASS |
REP-03 | Equal Allow rule wins the tie | ExampleBot/asset | ALLOW | ALLOW · PASS |
REP-04 | Duplicate product groups are combined | ExampleBot/two/page | DISALLOW | DISALLOW · PASS |
REP-05 | Wildcard and end anchor | ArchiveBot/reports/crawl.pdf | DISALLOW | DISALLOW · PASS |
REP-06 | Query component participates in matching | ExampleBot/article?preview=1 | DISALLOW | DISALLOW · PASS |
REP-07 | No applicable group allows access | ExampleBot/anything | ALLOW | ALLOW · PASS |
Inspect the exact fixtures
REP-01 — Exact product group replaces wildcard
User-agent: * Disallow: / User-agent: ExampleBot Allow: /public/
Decision trace: Allow line 5 is the most specific matching rule.
REP-02 — Longest matching rule wins
User-agent: ExampleBot Allow: /private/preview/ Disallow: /private/
Decision trace: Allow line 2 is the most specific matching rule.
REP-03 — Equal Allow rule wins the tie
User-agent: ExampleBot Disallow: /asset Allow: /asset
Decision trace: Allow line 3 is the most specific matching rule.
REP-04 — Duplicate product groups are combined
User-agent: ExampleBot Disallow: /one/ User-agent: ExampleBot Disallow: /two/
Decision trace: Disallow line 5 is the most specific matching rule.
REP-05 — Wildcard and end anchor
User-agent: * Disallow: /*.pdf$
Decision trace: Disallow line 2 is the most specific matching rule.
REP-06 — Query component participates in matching
User-agent: * Disallow: /*?preview=1$
Decision trace: Disallow line 2 is the most specific matching rule.
REP-07 — No applicable group allows access
User-agent: OtherBot Disallow: /
Decision trace: No exact product-token group or wildcard group exists, so no rules apply.
What this evidence can support
The fixtures verify deterministic group selection, exact-token fallback, repeated-group merging, longest-match precedence, equal-specificity Allow ties, wildcard and end-anchor handling, query matching, and the no-applicable-group default in the current implementation.
They do not verify a live robots.txt response, redirects, content type, caching, network failures, crawler identity, vendor extensions, or whether a crawler obeys the file. Those require a separately dated external observation.
Reproduction contract
- Use fixture set
REP-2026-08-22.1from the versioned source. - Run the repository build and QA; every expected decision must equal the actual result.
- Repeat one case in the public Robots.txt Rule Tester.
- Record tool version, input, product token, target, timestamp, output, and any parser warning.