AI Cost Firewall v0.7.0 — Controlled Streaming

AI Cost Firewall v0.7.0 adds controlled OpenAI-compatible streaming while preserving the same response controls, cache semantics, Privacy restoration, accounting, and evidence lifecycle used by non-streaming requests.

Highlights

• adds controlled stream=true support for OpenAI-compatible chat completions;
• consumes provider SSE internally and assembles a complete canonical response before client commit;
• converges ordinary JSON, assembled provider streams, and eligible cache hits on the same ChatCompletionResponse processing path;
• runs response Security Guard scanning on the complete response before delivery;
• restores Privacy Guard placeholders before approved SSE replay;
• uses transport-independent cache identity, allowing JSON-to-SSE and SSE-to-JSON cache reuse;
• supports stream_options.include_usage;
• reconstructs fragmented content, tool/function arguments, and compatible response metadata before replay;
• adds idle and absolute-generation timeout hardening for provider streams;
• expands stream metrics, evidence, Grafana diagnostics, examples, and integration tests.

Controlled response commit barrier

The core guarantee is:

No generated response content leaves AI Cost Firewall until AI Cost Firewall has approved the complete response.

On a cache miss, provider SSE is parsed and assembled internally. The canonical response then passes response controls, eligible cache storage, Privacy restoration, accounting, metrics, and evidence before AIF selects the final client transport.

ordinary upstream JSON ───────┐
                              ├─► canonical ChatCompletionResponse
assembled upstream stream ────┤
                              │
eligible cache hit ───────────┘
                                      │
                                      ▼
                              response Security Guard
                              eligible cache store
                              (cache miss only, pre-restore)
                              Privacy Guard restore
                              accounting / evidence
                                      │
                         ┌────────────┴────────────┐
                         ▼                         ▼
                  JSON serializer             SSE encoder / replay
                  stream=false                stream=true

Malformed, truncated, oversized, stalled, or otherwise failed provider streams return a normal HTTP error before downstream SSE is committed.

Configuration

Controlled streaming is enabled by default:

streaming_enabled true;
max_stream_upstream_bytes 8M;
upstream_timeout_seconds 120;

max_stream_upstream_bytes is the maximum cumulative provider SSE payload accepted for one controlled streaming request.

For controlled streams, upstream_timeout_seconds also acts as the maximum idle gap between provider SSE body chunks after headers are received. Controlled generation additionally has a fixed 15-minute absolute ceiling so continuous drip-feed data cannot hold an upstream concurrency permit indefinitely.

Provider SSE support is required only for stream=true; providers without streaming remain usable for ordinary JSON requests.

Observability

Streaming metrics include:

aif_stream_requests_total
aif_stream_completed_total
aif_stream_errors_total
aif_stream_aborted_total
aif_stream_upstream_errors_total
aif_stream_upstream_chunks_total
aif_stream_upstream_bytes_total
aif_stream_upstream_time_to_first_byte_seconds
aif_stream_generation_duration_seconds
aif_stream_client_time_to_first_byte_seconds
aif_stream_upstream_response_bytes
aif_stream_client_buffer_bytes
aif_stream_duration_seconds

The Grafana Diagnostics dashboard adds controlled-stream outcome, latency, upstream-intake, and payload-size panels.

Evidence

Controlled streaming adds stream lifecycle evidence such as:

upstream.stream.completed
upstream.stream.failed

Failure evidence records whether content was committed to the client. Request traces continue to contain exactly one terminal lifecycle event: request.completed or request.failed.

Upgrade notes

Use the v0.7.0 cumulative upstream SSE limit directive:

max_stream_upstream_bytes 8M;

Use upstream_timeout_seconds to control the maximum idle gap between provider SSE chunks. No additional VCAL module upgrade is required solely for AIF v0.7.0 controlled streaming.