Every major AI platform sends crawlers to fetch web content. Those crawlers feed the systems that generate the answers your buyers see. Understanding how they work is the foundation of any GEO or AI visibility strategy.
The landscape is not one type of bot. There are training crawlers that build a model's general knowledge, citation bots that look for a specific answer at the moment someone asks a question, and user-triggered fetchers that fire because a real person is waiting on a response right now.
This guide covers which crawlers actually visit, what they fetch and what they skip, why so many single-page apps look empty to them, and a practical checklist for making your site genuinely crawlable.
The crawlers actually visiting your site
Before optimizing anything, it helps to know who is showing up and for what purpose. These are the AI crawlers most sites see.
GPTBot (OpenAI, training)
GPTBot crawls pages to help train and update the models behind ChatGPT. It identifies itself with the user agent GPTBot, respects robots.txt, and does not execute JavaScript. It reads the raw HTML response and extracts text, headings, and any structured data it finds there.
ClaudeBot (Anthropic, training)
ClaudeBot fetches content for Anthropic's models. It also reads raw HTML with no JavaScript execution. Anthropic publishes its crawler IP ranges, which many sites use to verify the bot instead of relying on the user agent string alone.
OAI-SearchBot and PerplexityBot (citation, real time)
OAI-SearchBot and PerplexityBot work differently from a training crawler. They often fetch a page at the moment a user asks a question, looking for a specific, quotable answer to cite. There is no grace period here. If the page cannot be parsed in that fetch, the answer engine has nothing to cite from it.
ChatGPT-User, Claude-User, and Perplexity-User (user-triggered fetches)
These are not autonomous crawlers. They fire when a person asks ChatGPT, Claude, or Perplexity to open or read a specific page during a conversation. Technically they behave like the crawlers above (an HTTP fetch, no JavaScript), but they represent a real person waiting on an answer right now.
Google's AI crawler and the JavaScript exception
Google uses a separate crawler for its AI systems, distinct from classic Googlebot. Unlike most AI crawlers, Google's AI crawler and Apple's crawler are known to render JavaScript in at least some cases, similar to how Googlebot has long queued pages for rendering. That makes them the exception, not the rule. Do not assume any AI crawler will run your JavaScript.
Training bots vs citation bots: a different job entirely
The AI crawler landscape splits into two different jobs. Training crawlers like GPTBot and ClaudeBot fetch pages on their own schedule to build or refresh a model's underlying knowledge. What they read today may not show up in an answer for months, if it shows up at all.
Citation bots like OAI-SearchBot and PerplexityBot, along with user-triggered fetchers like ChatGPT-User and Claude-User, work at query time. They fetch a page because someone asked a question right now, and what they find in that single fetch becomes the source, or gets skipped, for that specific answer.
- Training bots reshape long-term knowledge. A page fixed today may not change a training-bot-fed answer immediately. These crawlers run on their own cycle.
- Citation bots decide the next answer. A page that is broken, blocked, or JavaScript-only at the exact moment a citation bot fetches it can lose that specific citation, even if the content is genuinely good.
- Both need the same foundation. Neither type benefits from content that only exists after a script runs. Fix the underlying rendering problem once and both crawler types benefit.
Why almost none of them execute JavaScript
An AI crawler is, at its core, an HTTP client. It sends a request, receives a response, and reads what came back. There is no browser engine underneath, no DOM, and no JavaScript runtime for GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot. Whatever your server returns in that first response is the whole page as far as the crawler is concerned.
This is different from how Googlebot has worked for years, where a separate rendering queue eventually executes JavaScript and revisits the page. Most AI crawlers do not have an equivalent second pass. What they get on the first fetch is what they get.
Why single-page apps and heavy client rendering look empty
A React, Vue, or similar client-rendered app typically ships an HTML shell such as a near-empty div and a script tag. A browser downloads that shell, runs the JavaScript, and builds the visible page. An AI crawler stops at the shell. Your headline, product description, and FAQ content never technically existed as far as that fetch was concerned.
Frameworks that support server rendering are not automatically safe either. A Next.js page built with server components or static generation sends full content in the first response. The same app with a page that fetches data client-side after the initial load behaves like a plain single-page app for that one route. Rendering mode is a per-page decision, not a per-framework guarantee.
Robots.txt and sitemaps: how crawlers decide what to fetch
Before an AI crawler reads your content, it usually checks robots.txt for that user agent. Most major AI crawlers respect these directives, so a Disallow rule genuinely keeps a compliant crawler out. That control cuts both ways. A rule copied from a template, or a managed robots.txt override added automatically by a CDN or hosting provider, can quietly block a citation bot you actually want to reach you.
Sitemaps help crawlers discover pages faster and prioritize what to fetch. An accurate, current sitemap that reflects your real URL structure reduces the odds that a page sits undiscovered for months.
- Check robots.txt for your actual AI crawler user agents. Not just Googlebot. Look for GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot specifically.
- Watch for CDN or hosting-level overrides. Some platforms inject their own robots.txt rules above your own, sometimes blocking citation bots by default without telling you.
- Keep your sitemap accurate. Remove dead URLs, add new pages promptly, and make sure the sitemap is reachable and referenced from robots.txt.
A practical checklist for making your site crawlable
- Fetch your own pages like a crawler. Run something like curl with an AI crawler user agent and read the raw response. If your headline and key facts are not in that output, an AI crawler cannot see them either.
- Serve core content in the first response. Server-side rendering or static generation puts your real text, headings, and FAQ content in the HTML a crawler actually receives, no JavaScript required.
- Keep structured data in the initial HTML. JSON-LD and schema markup in the head should render on the server, even on pages where some body content is client-rendered.
- Confirm robots.txt allows the crawlers you want. Verify this for the exact bot names, not just a generic wildcard rule, and recheck after any CDN or platform change.
- Keep your sitemap current and linked from robots.txt. This helps crawlers find new and updated pages without waiting on incidental discovery.
- Prioritize the pages that matter most. Homepage, product or service pages, and comparison content are the pages citation bots are most likely to fetch when someone asks a buying question.
FAQ: how AI crawlers work
Do AI crawlers like GPTBot execute JavaScript?
No, not the major ones. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot fetch raw HTML and do not run a JavaScript engine. Google's AI crawler and Apple's crawler are known exceptions that can render JavaScript in at least some cases, but you should not assume any AI crawler will run your scripts.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls on its own schedule to help train and update OpenAI's models. OAI-SearchBot fetches pages at the moment a user asks a question, looking for a specific answer to cite in that response. One shapes long-term knowledge, the other shapes the next answer.
How can I check what an AI crawler actually sees on my page?
Fetch the page yourself with a plain HTTP request using an AI crawler's user agent, for example with curl, and read the raw response. If your headline, key facts, or FAQ content are missing from that output, the crawler cannot see them either.
Can I block AI crawlers with robots.txt?
Yes. Most major AI crawlers respect robots.txt directives for their specific user agent. This also means an accidental Disallow rule, sometimes added automatically by a CDN or hosting platform, can block a citation bot without you realizing it.
Are training crawlers and citation bots the same thing?
No. Training crawlers like GPTBot and ClaudeBot build or refresh a model's general knowledge on their own schedule. Citation bots like OAI-SearchBot and PerplexityBot, along with user-triggered fetchers like ChatGPT-User, typically fetch a page in real time to support one specific answer.