Skip to main content

What is AI Cost Firewall?

AI Cost Firewall is a pilot-ready OpenAI-compatible API gateway for caching, cost control, and operational visibility.

It sits between your application and an upstream LLM provider. Applications send requests to AI Cost Firewall instead of calling the provider directly. The firewall checks whether a response can be reused from cache and forwards only necessary requests upstream.

It supports practical OpenAI-compatible upstream and embedding endpoints, including cloud APIs, local gateways, and self-hosted model servers.

v0.4.2 release focus​

AI Cost Firewall v0.4.2 adds asynchronous buffered delivery of structured evidence events to VCAL Audit.

The release builds on the evidence lifecycle and guard orchestration introduced in v0.4.0. AI Firewall can now send vcal.evidence.event schema version 1.1 batches to VCAL Audit through a bounded in-memory queue with configurable batching, flush intervals, request timeouts, retries, and backoff.

VCAL Audit can persist evidence in SQLite, reconstruct request traces by trace_id, export retained events, and verify an authoritative SHA-256 record hash chain.

Supported operating modes:

AI Firewall only
AI Firewall + VCAL Security Guard
AI Firewall + VCAL Privacy Guard
AI Firewall + both guard modules
AI Firewall + VCAL Audit
AI Firewall + guard modules + VCAL Audit

When both guard modules are enabled, AI Firewall coordinates:

Security Guard request scan
→ Privacy Guard anonymize/redact
→ Redis/Qdrant cache lookup or upstream LLM
→ Security Guard response scan
→ Privacy Guard restore

VCAL Security Guard, VCAL Privacy Guard, and VCAL Audit are optional commercial modules and are not required for standalone AI Firewall deployments.

Each request trace that emits request.received ends with exactly one request.completed or request.failed evidence event. Streaming requests are rejected globally with HTTP 422 before cache, guard, or upstream processing.

Why it exists​

LLM applications often send repeated or semantically similar prompts. Without caching, every request can become an upstream API call, token usage, added latency, and additional cost.

AI Cost Firewall reduces this waste with two cache layers:

  1. Exact cache — reuses responses for identical normalized requests.
  2. Semantic cache — reuses responses for similar prompts when similarity is high enough.

Core capabilities​

  • OpenAI-compatible /v1/chat/completions endpoint
  • Redis / Valkey exact caching
  • Qdrant semantic caching
  • Prometheus metrics and Grafana dashboards
  • strict configuration validation
  • model allowlist behavior through model_price
  • request size limits
  • readiness and liveness endpoints
  • graceful shutdown and hot reload via SIGHUP
  • semantic cache lifecycle control
  • ready-to-run Docker Compose deployment examples
  • OpenAI-compatible provider patterns for OpenAI, Ollama, LM Studio, vLLM, LiteLLM, and OpenRouter
  • /version endpoint for release and compatibility introspection
  • structured vcal.evidence.event schema v1.1 events
  • optional buffered HTTP evidence delivery to VCAL Audit
  • configurable Audit queue, batching, timeout, retry, and backoff behavior
  • stable trace correlation and terminal lifecycle evidence

Deployment examples​

Ready-to-run pilot deployment patterns are available under:

deploy/examples/

These include OpenAI cloud, local Ollama, hybrid OpenAI + local embeddings, OpenRouter, and a full local stack with dashboards.

  1. What it does
  2. Quick Start with Docker
  3. Request flow
  4. Configuration overview
  5. Runtime overview
  6. Metrics

v0.4.2 validation​

AI Firewall v0.4.2 was validated with VCAL Privacy Guard, VCAL Security Guard, and VCAL Audit. Validation covered guarded requests, buffered evidence delivery, trace reconstruction by trace_id, SQLite persistence, and successful SHA-256 record-chain verification.