Your firewall is your AI policy

01 / the gate nobody measures

Four things have to be true before any of the advice applies

The generative-engine-optimization industry sells schema markup, entity coverage, citation-friendly formatting. All of it sits downstream of four gates that almost nobody checks, in this order: the crawler has to reach you, it has to get an answer fast enough to stay, there has to be text in the HTML it received, and you have to have permitted the use.

The third gate is the one that surprises people. GPTBot, ClaudeBot and PerplexityBot do not execute JavaScript. If your content arrives via client-side rendering, those crawlers receive an empty shell — and you will not find out from your analytics, because a crawler that gets a useless page does not report anything.

So I built a probe that walks all four gates for a given domain, once per crawler, and pointed it at 18 sites whose behaviour I could not have guessed.

02 / the status codes

You can diff a company's AI strategy with curl

The Guardian has a content deal with OpenAI. Its server serves my GPTBot, OAI-SearchBot and ChatGPT-User requests a clean 200, and its robots.txt does not name them. The same request wearing ClaudeBot, PerplexityBot or CCBot gets a 403 — and those names are in the robots.txt disallow list too. Policy and enforcement agree.

The New York Times, currently in litigation with OpenAI, 403s almost the entire field: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Common Crawl, meta-externalagent. Two user-agents got through — bingbot and Amazonbot.

The business-development org and the infrastructure config have become the same document. Whatever a company says publicly about its posture toward AI, the WAF rule is the version that is actually in force, and it is readable by anyone with a terminal.

03 / policy versus enforcement

robots.txt is a sign on the door. The WAF is the lock.

Reddit's robots.txt blocks every AI crawler in the list — fourteen tokens, no exceptions. Enforcement tells a different story. My GPTBot request got a 403 and my ClaudeBot and CCBot requests got 429s, but OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot and meta-externalagent all received a 200 from the same IP, seconds apart. The training-oriented crawlers hit a wall; several of the retrieval-oriented ones walked through. Whether that is deliberate or a rule set that simply has not caught up with the names, I cannot tell from outside — but the policy and the enforcement are not the same policy.

Figma runs the inverse configuration. Its robots.txt disallows GPTBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, CCBot and Google-Extended — and its server returns 200 to every single one of them. That is a policy that exists entirely on the honour system, which is a legitimate choice, but worth knowing you have made.

04 / accidental invisibility

Some very famous sites are blank to AI by accident

Airbnb 403s AI crawlers at the WAF andserves 94 visible words of raw HTML — belt and suspenders, if what you are securing is invisibility. LinkedIn serves 23 words to anything without a session; the authwall applies to the machines writing answers about your career too. Reddit's homepage contains one visible word before JavaScript runs.

The sites doing it right are boringly consistent — full server-rendered HTML, fast, no bot discrimination:

† The probe's own baseline browser request was filtered on these five, so their rows describe my probe environment rather than their treatment of AI crawlers. See section 05.

Stripe is the high score: 1,957 words in raw HTML, a 200 for every crawler, 0.250s cold. Anthropic returned the fastest warm response in the set at 0.102s. MDN, Vercel and Shopify all land at 97 or above on the same recipe. None of this is exotic — it is server-rendered HTML on a fast origin with no bot rules that nobody remembers writing.

05 / what this cannot tell you

The two findings I had to delete

My first draft of this piece contained a detail I liked a great deal: that openai.com 403s a request claiming to be GPTBot, because they verify their own crawler by IP range and treat name-only claims as impostors. It is a good story. The data does not support it. openai.com also 403'd my baseline browser request, before any bot user-agent was involved — so I cannot distinguish “they detect GPTBot impostors” from “they challenge everything from a datacenter IP.” The second reading is simpler and I have no evidence against it.

The same draft said the tool withholds a content classification for the five filtered sites. It does not — it scores them and stamps a BASELINE_ANOMALY flag saying the results reflect probe-environment filtering rather than real bot experience. Accurate description of one's own tool turns out to be harder than building it.

This is why the probe is only half the tool. The other half reads server logs: what real crawlers actually received, with their source IPs verified against vendor-published ranges, which is the only way to answer the question the probe can merely raise. Log mode also surfaces the failure nothing else catches — 499s, the status your log records when a crawler gave up waiting on a slow origin and hung up. No error page, no alert, no citation.

06 / if you run websites

The unglamorous order of operations

Check reachability first: point something at your own domain with each crawler's user-agent and compare against a browser baseline. Bot management ships with defaults, defaults do not know your commercial intent, and a 403 to a retrieval crawler is a citation you will never appear in.

Then check the cold number rather than the warm one, because crawlers land on the pages your cache has never seen. Then count the words in curl output rather than in the rendered page. Then read your own robots.txt against your own WAF rules and find out whether they agree.

And no, llms.txt is not one of the gates. The tool checks it and weights it at zero: measured adoption data shows the overwhelming majority of llms.txt files receive no AI-crawler requests at all. It costs nothing to publish and it is not a substitute for being fetchable.