explainer: cloudflare ai gateway

ai gateway is a control plane and observability layer that sits in front of your ai calls. it does not host models. you point your app at the gateway, the gateway forwards to the actual provider (workers ai, openai, anthropic, google, aws bedrock, groq, mistral, etc), and gives you rate limiting, caching, logging, and fallback along the way.

→ /ai-chat is the live demo. this page just describes what it's doing.

# what ai gateway gives you

rate limiting	configurable per-gateway. this site uses 3 req / hour / ip (sliding). protects your api key + budget from abuse.
caching	identical prompts return the previous response without hitting the model. saves neurons and latency.
logs	every request/response captured with timing, cost estimate, cache hit/miss, and full body. searchable in the dashboard.
analytics	tokens consumed, cost, request volume, error rate — all charted, per model, per gateway.
fallback	if the primary provider errors or times out, route to a backup provider (or a smaller/cheaper model on the same provider).
evaluations	run test prompts against multiple models side-by-side and score them.
guardrails	content moderation, pii detection, prompt injection detection before/after the model call.

# how /ai-chat uses it

  ┌─────────┐  POST /api/chat        ┌─────────────┐  env.AI.run(model,
  │ browser │───────────────────────→│ pages func  │─  input, {gateway:{
  └─────────┘                        └──────┬──────┘  id:'site-ai-gateway'}})
                                            │
                                            ▼
                              ┌──────────────────────────┐
                              │  ai gateway              │  ← rate limit 3/hr/ip
                              │  id: site-ai-gateway     │  ← log request
                              │  provider: workers-ai    │  ← cache lookup
                              └──────────┬───────────────┘
                                         │ forward on miss
                                         ▼
                              ┌──────────────────────────┐
                              │  workers ai              │
                              │  llama-3.3-70b-...-fast  │
                              └──────────────────────────┘

# this gateway's settings

gateway id	`site-ai-gateway`
rate limit	3 requests / 3600s per ip, sliding window
cache ttl	0s (disabled — chat responses are stateful and short-lived)
log collection	enabled
upstream	workers ai (universal endpoint via the workers ai binding)

# when to layer this vs. skip it

if you're building a serious ai product, use ai gateway. the operational win (logs, rate limits, cost caps, provider swap without redeploying) is huge. if you're prototyping and calling one model from a single script, skip the layer until you actually feel one of its pains.