demo: workers ai + ai gateway chat
a small chat bot backed by llama-3.3-70b-instruct-fp8-fast on workers ai, routed through cloudflare ai gateway for rate limiting and observability. the model has no internet access, so it won't know today's weather or news. ask it about web tech, cloudflare products, networking, etc.
limits (to prevent abuse of my neuron budget):
3 requests per hour per ip (ai gateway).
turnstile challenge required.
500 char input max, 200 token output max.
off-topic questions get a polite decline.
# try one of these
# ask
waiting…
# how it works
- page function
/api/chatverifies turnstile server-side - truncates prompt to 500 chars, adds a system prompt (topic guardrail)
- calls
env.AI.run(...)with the AI Gateway option set tosite-ai-gateway - ai gateway enforces its own rate limit (3/hour/ip, sliding window)
- on 429, we surface a friendly message with a suggested retry time
# config
| model | @cf/meta/llama-3.3-70b-instruct-fp8-fast |
| ai gateway | site-ai-gateway (see /ai-gateway) |
| rate limit | 3 req / 3600s / ip, sliding |
| max tokens | 200 output, 500 char input |
| bindings | AI (workers ai), TURNSTILE_SECRET_KEY |