What AI Cost Firewall Does
AI Cost Firewall acts as a smart gateway for OpenAI-compatible chat-completion requests.
Instead of sending every request directly to an LLM provider, it checks whether the same or a similar request has already been answered.
Responsibilities
- validate requests
- normalize requests
- check exact cache
- check semantic cache
- forward upstream when needed
- store cache entries
- estimate token and cost savings
- expose Prometheus metrics and Grafana dashboards for cache diagnostics, cost visibility, and guard orchestration health
- handle readiness, shutdown, and reload behavior
- route cache misses to OpenAI-compatible upstreams
- emit schema-versioned evidence events with stable trace correlation
- optionally deliver evidence asynchronously to VCAL Audit through a bounded, batched HTTP sink
- reject streaming requests before cache, guard, or upstream processing
OpenAI-compatible gateway
Supported endpoint:
POST /v1/chat/completions
Existing applications can point their OpenAI-compatible client to AI Cost Firewall.
Two cache layers
| Layer | Backend | Purpose |
|---|---|---|
| Exact cache | Redis / Valkey | Reuse identical normalized requests |
| Semantic cache | Qdrant | Reuse semantically similar prompts |
Exact cache hits are fastest and have no embedding lookup cost. Semantic cache hits require embedding lookup but can reuse answers for similar prompts.
Deployment Patterns
AI Cost Firewall supports practical OpenAI-compatible deployment patterns including:
- OpenAI cloud deployments
- fully local Ollama deployments
- hybrid OpenAI + local embedding deployments
- OpenRouter routing deployments
- self-hosted vLLM deployments
Runnable examples are available under:
deploy/examples/
Optional enterprise guards
AI Cost Firewall v0.4.2 can orchestrate optional VCAL Security Guard and VCAL Privacy Guard modules.
Security Guard provides a deterministic, auditable first control layer for text-based risks such as prompt injection, jailbreak attempts, system-prompt extraction, unsafe tool-use instructions, data-exfiltration attempts, and common cyber-abuse patterns.
Privacy Guard can detect, redact, anonymize, and restore sensitive text values such as emails, IP addresses, phone-like values, API keys, bearer tokens, JWT-like values, private-key material, credit-card-like values, URL tokens, and configured custom patterns.
The guard modules are optional commercial add-ons. AI Cost Firewall remains usable as a standalone open-source caching and cost-control gateway.