Skip to main content

Architecture Overview

AI Cost Firewall is a lightweight LLM infrastructure component.

Components​

AI Cost Firewall​

The Rust + Axum gateway that validates requests, rejects unsupported streaming, checks caches, forwards misses upstream, stores cache entries, exposes metrics, and emits structured evidence events.

Redis / Valkey​

Stores exact cache entries.

Qdrant​

Stores semantic cache entries and performs vector search. AI Cost Firewall uses Qdrant gRPC on port 6334.

OpenAI-compatible chat upstream​

Receives cache misses.

OpenAI-compatible embedding provider​

Semantic caching uses embeddings generated by the configured embedding provider. The embedding provider may be the same service as the chat upstream or a separate OpenAI-compatible endpoint.

OpenAI-Compatible Provider Flexibility​

AI Cost Firewall supports practical OpenAI-compatible providers including:

  • OpenAI
  • Ollama
  • LM Studio
  • vLLM
  • LiteLLM
  • OpenRouter

without requiring provider-specific configuration blocks.

The deployment stack also includes:

  • Prometheus
  • Grafana
  • Overview dashboard
  • Diagnostics dashboard

See:

deploy/examples/

Prometheus and Grafana​

Prometheus scrapes /metrics; Grafana visualizes cache performance, savings, and diagnostics.

Enterprise guard orchestration​

AI Cost Firewall v0.4.2 can orchestrate optional VCAL enterprise modules while keeping the core gateway usable as a standalone caching layer.

Supported modes:

AI Firewall only
AI Firewall + VCAL Security Guard
AI Firewall + VCAL Privacy Guard
AI Firewall + VCAL Security Guard + VCAL Privacy Guard

Full enterprise flow:

Client
  -> AI Cost Firewall
      -> VCAL Security Guard request scan
      -> VCAL Privacy Guard anonymize/redact
      -> exact/semantic cache lookup or upstream LLM
      -> VCAL Security Guard response scan
      -> VCAL Privacy Guard restore
      -> Client

Ordering matters:

  • Security Guard blocks malicious prompts before privacy mapping, cache lookup, or upstream forwarding.
  • Privacy Guard anonymizes sensitive text before Redis, Qdrant, semantic cache payloads, or upstream providers.
  • Security Guard scans assistant output before Privacy Guard restore.
  • Privacy Guard restore is the final response transformation.

Evidence and audit boundary​

AI Cost Firewall v0.4.2 emits vcal.evidence.event schema v1.1 records to structured application logs. Each trace that emits request.received ends with exactly one request.completed or request.failed event.

Prometheus and the AI Firewall Grafana dashboards remain focused on live operational metrics. VCAL Privacy Guard and VCAL Security Guard keep their own module-specific dashboards. VCAL Audit is available as a separate consumer for retained trace search, lifecycle analysis, evidence export, and SIEM/SOAR integration.

Buffered Audit delivery​

AI Firewall v0.4.2 can route structured evidence to a buffered HTTP sink:

request processing
  -> evidence event
  -> bounded in-memory queue
  -> batch by size or flush interval
  -> POST /v1/events/batch
  -> VCAL Audit

The Audit path is asynchronous. Temporary Audit failures do not normally fail the client request. Retry exhaustion can result in evidence loss because the current sender queue is memory-backed.