demo: block ai scrapers at the edge

a waf custom rule blocks known ai crawler user-agents (gptbot, claudebot, perplexitybot) on /api/ai-probe. because browsers refuse to let javascript spoof user-agent, the rule also matches the demo header x-test-bot: gptbot so we can trigger the block from the button below.

# method 1: curl (any terminal)

# spoof a known ai crawler UA
curl -i -A "GPTBot/1.0" "https://rhocfsandbox.com/api/ai-probe"

# other UAs the rule matches
curl -i -A "ClaudeBot/1.0" "https://rhocfsandbox.com/api/ai-probe"
curl -i -A "PerplexityBot/1.0" "https://rhocfsandbox.com/api/ai-probe"

# a normal browser UA passes through → 200
curl -i -A "Mozilla/5.0" "https://rhocfsandbox.com/api/ai-probe"

# method 2: in-page fetch (this button)

fires fetch() with the alternate header. browsers won't let us set user-agent, so the rule matches on x-test-bot for the browser flow.

click a button…

# what's the rule

rule name	`SITE_ai_scraper_block`
expression	`http.request.uri.path eq "/api/ai-probe" and (lower(http.user_agent) contains "gptbot" or lower(http.user_agent) contains "claudebot" or lower(http.user_agent) contains "perplexitybot" or any(http.request.headers["x-test-bot"][*] contains "gptbot"))`
action	block · custom html response (403)

cloudflare also ships a zone-wide managed toggle ("block ai scrapers and crawlers") that maintains this list for you. for the demo we use a scoped custom rule so it only fires on the demo endpoint.