How AI Assistants Choose Which Brands and Sources to Cite

# ai# seo# marketing# python
How AI Assistants Choose Which Brands and Sources to CiteOmar Eldeeb

What OpenAI, Perplexity, Anthropic and Google document about how their assistants pick sources: web search, crawlers, robots.txt. Plus a crawler check script.

AI assistants answer from two places: what the model learned in training and, when they search the web, the pages the search returns, which they show as citations. Vendors document the crawlers and controls (robots.txt tokens, location, search settings) but not how they rank sources, so treat any "ranking factor" list as a guess unless it cites the vendor.

This page sticks to what OpenAI, Perplexity, Anthropic and Google say in their own documentation, marks the places where the docs are silent, and gives you a script to check whether these assistants' crawlers may read your site.

Two ways an answer gets built

Answer from model knowledge Answer with web search
Where brands come from Patterns the model learned in training Pages the search step retrieved for this question, plus what the model already knows
Sources shown Usually none Citations or a sources list
How current As of the model's training data As current as the pages retrieved
What you can influence Slowly: being widely and consistently described across the web Being on the pages that get retrieved, and letting the search crawler read your site

Whether a given answer searched is often the model's own choice:

  • ChatGPT can search the web on its own when a question benefits from current information, and you can also turn search on yourself; answers that used search show their sources (OpenAI help: searching the web with ChatGPT).
  • OpenAI's API says the same about its web search tool: "the model can choose to search the web or not based on the content of the input prompt" (OpenAI web search guide).
  • Perplexity says it uses AI "to search the internet in real-time" and that "each answer includes numbered citations linking to the original sources" (Perplexity help center).

That's why measuring visibility means recording, per answer, whether search was used.

Which pages the search step can use

Each vendor runs separate crawlers for search and for training, with separate robots.txt tokens. From their own documentation:

Vendor Token What the vendor says it's for
OpenAI OAI-SearchBot "used to surface websites in search results in ChatGPT's search features." "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
OpenAI ChatGPT-User Certain user actions in ChatGPT and Custom GPTs; "not used for crawling the web in an automatic fashion", and "robots.txt rules may not apply."
OpenAI GPTBot Crawling content that "may be used in training our generative AI foundation models."
Perplexity PerplexityBot "designed to surface and link websites in search results on Perplexity"; "not used to crawl content for AI foundation models."
Perplexity Perplexity-User Fetches pages for user questions; "generally ignores robots.txt rules."
Anthropic Claude-SearchBot "navigates the web to improve search result quality for users"; blocking it "prevents our system from indexing your content for search optimization."
Anthropic Claude-User Retrieves content in response to a user query; blocking it prevents that.
Anthropic ClaudeBot Training: blocking it signals that future materials "should be excluded from our AI model training datasets."
Google Google-Extended Controls use for training future Gemini models and for grounding in Gemini Apps; it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

OpenAI adds that "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results.

The practical point: blocking a training crawler doesn't remove you from AI search, and blocking a search crawler can. If you want to be cited in ChatGPT search answers, OpenAI's documentation says OAI-SearchBot must be allowed.

Check your own site 

import sys
import urllib.robotparser

import requests

# User-agent tokens from each vendor's own crawler documentation.
AGENTS = {
    "OAI-SearchBot": "ChatGPT search results",
    "ChatGPT-User": "ChatGPT fetching a page for a user",
    "GPTBot": "OpenAI model training",
    "PerplexityBot": "Perplexity search results",
    "Claude-SearchBot": "Claude search indexing",
    "Claude-User": "Claude fetching a page for a user",
    "ClaudeBot": "Anthropic model training",
    "Google-Extended": "Gemini training and grounding (not Google Search)",
}


def check(site, path="/"):
    url = f"https://{site}/robots.txt"
    r = requests.get(url, timeout=20, headers={"User-Agent": "robots-check/1.0"})
    rp = urllib.robotparser.RobotFileParser()
    rp.parse(r.text.splitlines() if r.status_code == 200 else [])
    print(f"{site}{path}  (robots.txt HTTP {r.status_code})")
    for agent, purpose in AGENTS.items():
        verdict = "allowed" if rp.can_fetch(agent, f"https://{site}{path}") else "BLOCKED"
        print(f"  {agent:<17} {verdict:<8} {purpose}")


for site in sys.argv[1:]:
    check(site)
Enter fullscreen mode Exit fullscreen mode

python ai_robots.py datatooly.xyz on 2026-10-14:

datatooly.xyz/  (robots.txt HTTP 200)
  OAI-SearchBot     allowed  ChatGPT search results
  ChatGPT-User      allowed  ChatGPT fetching a page for a user
  GPTBot            allowed  OpenAI model training
  PerplexityBot     allowed  Perplexity search results
  Claude-SearchBot  allowed  Claude search indexing
  Claude-User       allowed  Claude fetching a page for a user
  ClaudeBot         allowed  Anthropic model training
  Google-Extended   allowed  Gemini training and grounding (not Google Search)
Enter fullscreen mode Exit fullscreen mode

Pass your own domain. Python's urllib.robotparser follows the basic robots.txt rules but not every extension, so treat a surprising result as a reason to read the file yourself. A missing robots.txt (HTTP 404) means everything is allowed. Your CDN or firewall can also block these crawlers regardless of robots.txt; check its bot settings too.

Other inputs the vendors document

  • Location. Answers to local questions depend on where the asker is. The OpenAI API's web search tool accepts an approximate user_location (country, city, region, timezone). The same "best dentist" question has different right answers in different cities.
  • Personalisation. Memory, chat history and custom instructions can shape answers in ChatGPT. OpenAI's shopping help article lists memory and custom instructions among the context it uses to pick products.
  • Product results aren't ads. The same shopping article says product results are chosen independently and that ads are shown separately.
  • How much the search step reads. In OpenAI's API, search_context_size changes how much retrieved material the model sees, and the docs say it "does not set an exact token count or guarantee a specific number of sources or citations."

What the vendors don't say

None of these vendors publishes how it ranks retrieved pages, how it picks which brands to name from them, or how training data shapes recommendations. Advice that lists precise "ranking factors" for ChatGPT or Perplexity is inference from experiments, not documentation. Some experiments are useful, but read them as observations that can change with the next model update.

What you can do with confidence follows from the documented parts:

  1. Let the search crawlers in (the table above), and check your CDN isn't blocking them.
  2. Find out which pages get cited for your buyer questions. Log the cited domains for every answer, as in How to check if ChatGPT mentions your brand. Those pages are what the search step currently reads for your category, whether they are yours, a competitor's, a review site or a forum.
  3. Make the facts easy to retrieve. Searched answers quote prices, plans and features from pages. Out-of-date pricing on your own site or a well-cited review page is a likely source of wrong answers about you.
  4. Measure, change one thing, and measure again with the same prompt set. Without a fixed baseline you can't tell a real gain from run-to-run noise; see AI share of voice for sample sizes.

Related guides

Do it without code

The free LLM visibility query builder turns your brand, category and competitors into a ready-to-run audit input, with a fixed example of the output. It doesn't call any AI model from your browser.

At scale

LLM Visibility Monitor on Apify records the sources behind each answer. It classifies every cited domain as owned, competitor, review site, Reddit or other user content, news, or other, ranks them, and lists the third-party sources cited in answers where a competitor appeared and you didn't. It also quotes the prices, plans and facts the answers state about your brand so you can check them. It asks the API models behind ChatGPT, Claude, Gemini, Perplexity and Grok and labels whether each answer used live web search; API answers can differ from the consumer apps. It's free to start, then pay-as-you-go.

Disclosure: I build the datatooly query builder and the Apify actor. Everything above is from the vendors' public documentation.

Sources: OpenAI: overview of OpenAI crawlers, OpenAI: web search tool guide, OpenAI Help Center: Searching the web with ChatGPT, Perplexity: How does Perplexity work?, Perplexity crawlers, Anthropic: crawlers and how to block them, Google's common crawlers (Google-Extended).