GPTBot (OpenAI/ChatGPT)
The most aggressive AI crawler, averaging 340 requests per day to optimized domains. Fetches raw HTML only. Respects robots.txt. Powers ChatGPT's real-time browsing.
Guides
How to Get Indexed by AI.
Make your website visible to AI crawlers.
AI indexing is fundamentally different from Google indexing. Each major AI platform has its own crawler — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini) — and each has different capabilities, preferences, and limitations. Getting indexed by all of them requires understanding how each one works and ensuring your site serves them usable content.
The AI crawler landscape
ClaudeBot (Anthropic)
Second most active at 180 requests per day. Focuses heavily on structured data and clean HTML. Powers Claude's web access.
PerplexityBot
Averaging 120 requests per day. Strong preference for FAQ-structured content and citation-ready formats. Powers Perplexity's real-time search.
Google-Extended (Gemini)
Growing rapidly at 200 requests per day. Benefits from existing Googlebot infrastructure. Powers Gemini and Google AI Overviews.
How to ensure AI indexing
Configure robots.txt correctly
Explicitly allow all AI crawlers. Many default robots.txt configurations block AI bots. Add allow rules for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
Serve complete HTML
AI crawlers don't execute JavaScript. Your page content must be in the initial HTML response. Server-side rendering, static generation, or adaptive rendering (Appear) ensures AI crawlers receive complete content.
Create an XML sitemap
AI crawlers use sitemaps for URL discovery. Ensure your sitemap covers all important pages and is referenced in robots.txt.
Deploy Appear for automatic optimization
Appear detects all 200+ AI crawler signatures in real time and serves each one structured, optimized content. One DNS change handles robots.txt compliance, content optimization, and structured data injection for every AI platform simultaneously.
Related guides
How to Appear in AI Search
Once indexed, make sure you appear in results.
Read the guideHow to Get Recommended by AI
Move from indexed to recommended.
Read the guideBest AI Visibility Tools
Tools that automate AI indexing.
Read the guideReady to get indexed by AI?
Appear deploys via a single DNS change and starts optimizing AI crawler responses immediately. No code changes, no redesigns, no engineering lift.
Install in minutes. Keep your stack. Improve AI visibility.