Engine v2.1·RFC 9309 §2.2.1 / §2.2.2·Deterministic SSR Analysis

GEO Auditor: Free AI Search & RFC 9309 Readiness Tool

Inspect HTTP headers, RFC 9309 AI crawler groups (OAI-SearchBot, Claude-SearchBot, PerplexityBot), initial SSR HTML delivery, and Schema.org graphs.

Format: https://example.com/page
✓ RFC 9309 Robots Validation✓ Bounded Stream 2MB✓ Zero Paid LLM Calls✓ DNS-Pinned Transport

Product policy: direct IP-literal URLs and non-standard ports are rejected in this MVP. Only public HTTP/HTTPS HTML/XHTML pages are scanned. Reports are private by default; you choose whether to create a shareable link.

Deterministic Verification Architecture

Engine v2.1 Methodology & 5 High-Precision Diagnostic Checks

GEO Auditor v2.1 evaluates 100 nominal points across four reproducible technical pillars—Crawler Accessibility & Protocol Compliance (30 pts), Initial SSR Content Delivery & Extraction (30 pts), Structured Data & Entity Graph Connectivity (25 pts), and Indexing Hygiene (15 pts)—plus five primary-source-verified 2026 diagnostics:

Check 01 · RFC 9309 §2.2.1 & §2.2.2

Specific-Group Override & Longest-Octet Check

Detects when a specific named group (e.g., User-agent: OAI-SearchBot or GPTBot) overrides User-agent: * (§2.2.1), resolves path conflicts by UTF-8 octet length with Allow winning ties (§2.2.2), and distinguishes 4xx allow (§2.3.1.3) from 5xx complete disallow (§2.3.1.4).

Check 02 · Anthropic Crawler Compliance

Deprecated Anthropic Token Trap Detector

Warns when a robots.txt file blocks legacy tokens (anthropic-ai or Claude-Web) while omitting ClaudeBot, Claude-SearchBot, or Claude-User (all three of which honor robots.txt and Crawl-delay).

Check 03 · ChatGPT Search Retrieval

OpenAI Labrador ~202-Character Snippet Simulator

Extracts the target page's full <title> and first 202 characters of visible text starting at the <h1> (skipping <nav> and ignoring meta description) to preview what ChatGPT Search's Labrador index stores.

Check 04 · Edge WAF Governance

Cloudflare Sept 15, 2026 'Block' vs. 'Disallow AI Training'

Checks whether the target URL returns 403/Challenge to search crawlers versus training crawlers and warns against Cloudflare's Sept 15, 2026 Training 'Block' trap that also blocks Googlebot, Bingbot, and Applebot.

Check 05 · Informational Scoping (0% Penalty)

/llms.txt & Regional Crawlers (Baidu/Naver/Yandex)

Reports /llms.txt as an unweighted coding-agent signal (0% search citation penalty) and evaluates Baiduspider, ERNIEBot, Yeti, and YandexAdditionalBot under Regional / Informational Scope.

CI/CD & Localhost CLI

geo-auditor-cli (MIT License · htmlparser2)

Execute deterministic readiness gates in GitHub Actions or against http://localhost:3000 with npx geo-auditor-cli <url> --fail-on-score 85 --json.

Definitive 2026 Multi-Engine Matrix (v4.0)

2026 Multi-Engine Crawler & Retrieval Matrix

Separates Anthropic's Claude-User (honors robots.txt) from ChatGPT-User and Perplexity-User, documents OpenAI's Labrador 202-character snippet behavior, reflects Google AI Overviews and AI Mode on Gemini 3.5 Flash / 3.x Pro, and scopes regional crawlers accurately.

2026 Multi-Engine Crawler and Retrieval Matrix detailing tokens, operators, operational classes, mechanics, and GEO Auditor actions
Token / AgentOperator & SurfaceOperational ClassVerified 2026 Mechanics & VerificationEngine v2.1 Rubric & Site Action
OAI-SearchBotOpenAI (ChatGPT Search / 'Labrador' index)Search / Citation CrawlerPopulates OpenAI's proprietary vertical indexes ('Labrador', 88.1% of May–July 2026 telemetry). Stores full <title>, URL, and ~202 chars of visible text starting at H1 (ignores meta description). Honors robots.txt (~24h). Verify via openai.com/searchbot.json. Does not execute JS (Vercel/MERJ).ALLOW in robots.txt + Cloudflare AI Crawl Control ('Search' = Allow). Ensure first 200 chars from H1 summarize tool clearly.
bingbot + IndexNowMicrosoft (Bing + Copilot + ChatGPT Deep Research)Primary Search & Grounding IndexSole index feeding Microsoft Copilot and used by ChatGPT Deep Research / Microsoft Web IQ grounding. IndexNow notifies Bing, Yandex, Naver, Seznam, Yep (NOT Google).ALLOW + automate IndexNow POST on deploy + track Bing Webmaster Tools 'AI Performance Report' (grounding queries).
GooglebotGoogle (Search + AI Overviews + AI Mode)Primary Search & WRS CrawlerExecutes JS via evergreen Chromium WRS (can be deferred). Both AI Overviews (2.5B+ MAU) and AI Mode (1B+ MAU, query fan-out) route via Gemini 3.5 Flash by default and escalate complex queries to Gemini 3.x Pro.ALLOW + verify GSC Settings > 'Search generative AI' is ON + monitor GSC Generative AI report (June 3 / Aug 31, 2026).
Claude-SearchBotAnthropic (Claude Web Search)Search / Citation CrawlerIndexes pages for Claude web search citations. Honors robots.txt & Crawl-delay. Verify IPs via claude.com/crawling/bots.json. (JS execution unmeasured in 2024 Vercel study; assume raw HTML).ALLOW in robots.txt + Cloudflare ('Search' = Allow). Flag deprecated tokens (anthropic-ai, Claude-Web) in Engine v2.1.
PerplexityBotPerplexity (Perplexity Search)Search / Citation CrawlerHonors robots.txt (~24h). Empirically observed not executing client JS (Vercel/MERJ study). Verify IPs via perplexitybot.json.ALLOW in robots.txt + Cloudflare ('Search' = Allow). Serve complete initial SSR HTML.
Claude-UserAnthropic (User-Triggered Fetch)User-Directed Page FetcherTriggered when a user asks Claude to visit a URL. UNLIKE OpenAI/Perplexity, Anthropic docs explicitly state Claude-User HONORS robots.txt disallow rules. Verify via claude.com/crawling/bots.json.ALLOW in both robots.txt and Cloudflare ('Agent' = Allow) so live Claude user prompts can inspect your URLs.
ChatGPT-User / Perplexity-UserOpenAI / PerplexityUser-Directed Page FetchersTriggered by human prompt. OpenAI states ChatGPT-User is 'not bound by robots.txt'; Perplexity states Perplexity-User 'generally ignores robots.txt'. Governed at WAF/IP layer.ALLOW at Cloudflare WAF / AI Crawl Control ('Agent' = Allow). Verify via openai.com/chatgpt-user.json & perplexity-user.json.
GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgentOpenAI, Anthropic, CommonCrawl, ByteDance, MetaFoundation Model Training CrawlersHonor robots.txt. Collect training corpora. WARNING: On Cloudflare (since Sept 15, 2026), choosing 'Block' on Training also blocks Googlebot/Bingbot! Use 'Disallow AI Training' instead.OPTIONAL (Set Cloudflare Training to 'Allow' or 'Disallow AI Training' — NEVER 'Block').
Google-Extended / Applebot-ExtendedGoogle / AppleControl Tokens (Not crawlers)Googlebot/Applebot perform the fetch. Google-Extended controls Gemini model training AND grounding in standalone Gemini Apps (does NOT affect Google Search, AIO, or AI Mode).ALLOW Google-Extended; Applebot-Extended is an optional clean opt-out for Apple Intelligence training.
YandexBot / YandexAdditionalBot / Yeti (Naver) / Baiduspider / ERNIEBotYandex (Alice AI) / Naver (AI Briefing) / Baidu (ERNIE)Regional Search & AI CrawlersYandex: Indexed sites are in Alice AI by default; YandexAdditionalBot is opt-out only. Naver: Yeti feeds AI Briefing (~70% citations from Naver UGC; host-only registration). Baidu: Verifies via rDNS (*.baidu.com / *.baidu.jp); uses Baiduspider & ERNIEBot/YiyanBot.Engine v2.1 reports these as 'Informational / Regional Scope (0% penalty on English sites)'. Leave allowed in robots.txt.
Pre-Rendered Benchmark Sample

Reference Audit Telemetry (Engine v2.1)

Pre-rendered deterministic inspection of /tools/geo-auditor verifying 5/5 search crawler allow rules, 198-character Labrador H1+lead snippet, and connected Schema.org entity graph:

{
  "engine": "2.1.0",
  "url": "https://www.kuzmanovicaleksandar.com/tools/geo-auditor",
  "score": 100,
  "confidence": "High",
  "pillars": {
    "crawlerAccessibility": "30/30 (5/5 Search Tokens Allowed, RFC 9309 §2.2.1)",
    "contentDelivery": "30/30 (SSR HTML Complete, <main> Landmark)",
    "structuredData": "25/25 (WebApplication + FAQPage + BreadcrumbList)",
    "indexingHygiene": "15/15 (Canonical Match, Heading Hierarchy)"
  },
  "labradorSnippet202": {
    "status": "Optimal",
    "charCount": 198,
    "startsAtH1": true,
    "containsFullH1": true
  },
  "deprecatedAnthropicTrap": false
}
CI/CD & Localhost CLI

Automate Checks with geo-auditor-cli

Published under the MIT license using htmlparser2 for DOM and Schema.org verification. Gate deployments in GitHub Actions or test localhost builds:

# Audit production or staging URL with score threshold
npx geo-auditor-cli https://www.kuzmanovicaleksandar.com/tools/geo-auditor --fail-on-score 85 --json

# Audit localhost server in CI
npx geo-auditor-cli http://localhost:3000 --fail-on-score 85 --json
Retrieval-Ready Technical Reference

Frequently Asked Questions (RFC 9309, Labrador & AI Search Readiness)

How does OpenAI's Labrador index extract page snippets for ChatGPT Search?
Telemetry from May 21 to July 21, 2026 (analyzed across 534 pages by Resoneo and Peec AI on September 4, 2026) showed that OpenAI's proprietary 'Labrador' retrieval index serves 88.1% of ChatGPT Search lookups. Unlike traditional search engines that rely on <meta name="description">, Labrador stores the full HTML <title>, the URL, and a ~202-character snippet from the start of visible text beginning at the <h1> (containing the H1 83.6% of the time). Engine v2.1 includes a Labrador ~202-Character Snippet Simulator to preview this exact window.
Does Anthropic's Claude-User honor robots.txt disallow rules?
Yes. Unlike OpenAI's ChatGPT-User (which OpenAI documents as 'not bound by robots.txt') and Perplexity's Perplexity-User (which 'generally ignores robots.txt'), Anthropic's crawler help center (updated April 7 and October 2026) confirms that all three Anthropic bots—ClaudeBot, Claude-SearchBot, and Claude-User—honor robots.txt disallow rules and Crawl-delay. Additionally, blocking deprecated tokens (anthropic-ai, Claude-Web) is ignored by current Anthropic crawlers under RFC 9309 §2.2.1.
How does RFC 9309 resolve User-agent group precedence and Allow/Disallow path conflicts?
Under RFC 9309 §2.2.1 (User-Agent Line), crawlers match User-agent groups case-insensitively and a specific named group completely overrides the wildcard User-agent: * group rather than merging with it. Within the active group, RFC 9309 §2.2.2 (Allow and Disallow Lines) resolves path conflicts by longest match measured in canonical UTF-8 octets, with Allow winning ties of equal octet count. Under §2.3.1.3, HTTP 4xx on /robots.txt permits all crawling, whereas §2.3.1.4 requires crawlers to assume complete disallow on HTTP 5xx.
Why does Cloudflare's September 15, 2026 AI Crawl Control 'Block' setting affect Googlebot and Bingbot?
Cloudflare replaced its legacy 'Block AI bots' toggle with Search, Training, and Agent policies on July 1, 2026, and on September 15, 2026 changed the 'Block' action on Training so that it also blocks mixed-use crawlers (Googlebot, Bingbot, and Applebot) at the WAF layer. Site owners who want to opt out of AI training without blocking search engines must select 'Disallow AI Training' (never 'Block') and verify HTTP 200 responses in Cloudflare Dashboard -> AI Crawl Control ('Crawlers' tab).
Did Google merge AI Overviews and AI Mode into a single feature in 2026?
At Google I/O on May 19, 2026, Google unified both surfaces onto Gemini 3.5 Flash by default (escalating complex queries to Gemini 3.x Pro). However, as documented in Google Search Console Help, AI Overviews (inline SERP synthesis serving 2.5B+ MAU) and AI Mode (conversational tab with query fan-out serving 1B+ MAU) remain two distinct search features. Both are gated by standard Googlebot indexing and the Google Search Console Settings -> 'Search generative AI' inclusion toggle.
Are /llms.txt and regional AI crawlers (Baidu, Naver, Yandex) required for English AI search citations?
No. Engine v2.1 scopes /llms.txt as an unweighted coding-agent discovery signal (0% search citation penalty) and reports Baiduspider, ERNIEBot, Yeti (Naver AI Briefing, which draws ~70% of citations from Naver UGC), YandexBot, and YandexAdditionalBot under 'Regional / Informational Scope (0% penalty on English sites)'. Similarly, blocking foundation model training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent) via robots.txt incurs zero penalty under the Search Discoverability profile.