Fetching, indexing and being cited in an answer are three different events. Only the first can be tested directly. Everything after that depends on the search or AI system.
Start by separating the three events
When a bot fetches your page, you initially know only this: one request received a server response. It does not prove that the URL was stored, indexed or later selected as a source.
Bot names also represent different jobs. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for possible model training. ChatGPT-User fetches a page on a user's behalf. Google applies its normal Search eligibility rules to AI Overviews and AI Mode; Google-Extended is a control token for certain Gemini uses, not a separate HTTP crawler.
Can a crawler reach the page at all?
Test the canonical HTTPS URL. It should respond consistently, normally with status 200. Then inspect robots.txt, meta robots and X-Robots-Tag separately. A robots.txt allowance is not enough when a CDN, firewall or bot defence still rejects the request.
Important answers and internal links should also be present in the delivered HTML. A page can look complete in a browser while exposing little more than an empty JavaScript shell to a crawler.
Different bots need different decisions
For ChatGPT search eligibility, OAI-SearchBot must not be blocked. Whether GPTBot may fetch content for training is a separate choice. Anthropic also separates search, training and user-triggered retrieval. Googlebot remains the relevant crawler for Google Search.
A user-agent is only a claim, though. For consequential decisions, use published IP ranges or the reverse and forward DNS procedure described by the operator.
A reachable page is not automatically a useful source
Give the page one clear job. Answer the real question early, define unfamiliar terms and support changing facts with primary sources. Original measurements, reproducible examples, screenshots or test data are often more useful than another broad summary.
- One URL, one primary user job and a matching H1
- Internal links from relevant pages — not merely a sitemap entry
- No series of near-identical pages for tiny keyword variations
Logs replace guesswork with evidence
A URL check shows what happens during one controlled fetch. To learn whether a particular crawler actually visited, inspect server, CDN or firewall logs. Look for its published token, verify the identity when it matters, and compare the path, status code and timestamp.
Webmaster tools add another layer. Search Console can show Google's indexing state; Bing provides its own Webmaster Tools. A robots.txt allowance or sitemap does not guarantee inclusion by OpenAI or Anthropic.
Shortcuts that do not solve the problem
An llms.txt file, special AI markup or hundreds of generated question pages do not replace crawl access and useful main content. Google does not require special AI schema for its AI search features. Structured data is useful only when it matches visible content and a supported search feature.
One mention in a test query is not a stable visibility score either. When observing AI answers, record the provider, prompt, date, region and cited sources. That leaves you with a useful sample instead of a misleadingly precise metric.
Check one URL in five minutes
These questions are enough for a first technical pass:
- Does the canonical HTTPS URL return a useful response?
- Are the heading, core answer and important links present in the initial HTML?
- Do robots.txt and page directives allow the intended crawler?
- Do relevant internal links and the sitemap expose this URL?
- Do logs later show real requests rather than only an allowed bot name?