Seven rules, exact inputs, executable results.
These cases test AnalyseSpider's local RFC-style evaluator. They do not benchmark an external crawler or prove that a bot obeys robots.txt.
Executed result matrix
| Fixture | Rule job | Token and target | Expected | Actual |
|---|---|---|---|---|
REP-01 | Exact product group replaces wildcard | ExampleBot/public/report | ALLOW | ALLOW · PASS |
REP-02 | Longest matching rule wins | ExampleBot/private/preview/index.html | ALLOW | ALLOW · PASS |
REP-03 | Equal Allow rule wins the tie | ExampleBot/asset | ALLOW | ALLOW · PASS |
REP-04 | Duplicate product groups are combined | ExampleBot/two/page | DISALLOW | DISALLOW · PASS |
REP-05 | Wildcard and end anchor | ArchiveBot/reports/crawl.pdf | DISALLOW | DISALLOW · PASS |
REP-06 | Query component participates in matching | ExampleBot/article?preview=1 | DISALLOW | DISALLOW · PASS |
REP-07 | No applicable group allows access | ExampleBot/anything | ALLOW | ALLOW · PASS |
Exact fixtures and decision trace
REP-01 — Exact product group replaces wildcard
User-agent: * Disallow: / User-agent: ExampleBot Allow: /public/
Decision trace: Allow line 5 is the most specific matching rule.
REP-02 — Longest matching rule wins
User-agent: ExampleBot Allow: /private/preview/ Disallow: /private/
Decision trace: Allow line 2 is the most specific matching rule.
REP-03 — Equal Allow rule wins the tie
User-agent: ExampleBot Disallow: /asset Allow: /asset
Decision trace: Allow line 3 is the most specific matching rule.
REP-04 — Duplicate product groups are combined
User-agent: ExampleBot Disallow: /one/ User-agent: ExampleBot Disallow: /two/
Decision trace: Disallow line 5 is the most specific matching rule.
REP-05 — Wildcard and end anchor
User-agent: * Disallow: /*.pdf$
Decision trace: Disallow line 2 is the most specific matching rule.
REP-06 — Query component participates in matching
User-agent: * Disallow: /*?preview=1$
Decision trace: Disallow line 2 is the most specific matching rule.
REP-07 — No applicable group allows access
User-agent: OtherBot Disallow: /
Decision trace: No exact product-token group or wildcard group exists, so no rules apply.
Boundary
The cases verify group selection and matching in this implementation. They do not verify a live response, caching, crawler identity, vendor extensions or compliance by an external bot.