Skip to main content

Troubleshooting

Confirm the running release​

Before debugging a pilot deployment, confirm the running binary and compatibility model:

curl -s http://localhost:8080/version

For v0.4.2, the response should report the running 0.4.2 release, supported_api_style as openai_compatible, and provider_specific_config_blocks as false.

Low cache hit rate​

Check:

aif_cache_exact_hits
aif_cache_semantic_hits
aif_cache_misses
aif_semantic_threshold_results_total{result="fail"}

Common causes: threshold too high, prompts not similar, semantic cache disabled, retention too short.

Startup diagnostics fail before the server is ready​

Check the firewall logs first. v0.4.2 startup diagnostics should identify whether the failure is related to configuration, Redis, Qdrant, vector-size compatibility, upstream connectivity, embedding provider behavior, DNS, connection errors, or TLS/certificate validation.

Use /healthz, /readyz, and /version together:

curl -i http://localhost:8080/healthz
curl -i http://localhost:8080/readyz
curl -s http://localhost:8080/version

/healthz only shows that the process is alive. /readyz shows whether it is ready to serve traffic. /version confirms what release is running.

Redis connection failure​

Docker Compose:

redis_url redis://redis:6379;

Local source run:

redis_url redis://127.0.0.1:6379;

Qdrant connection failure​

Docker Compose:

qdrant_url http://qdrant:6334;

Local source run:

qdrant_url http://127.0.0.1:6334;

Qdrant vector size mismatch​

Fix by recreating the collection, using a matching embedding model, or updating qdrant_vector_size.

ai-firewall: command not found​

After source build, use:

./target/release/ai-firewall

Or install it:

sudo install -m 0755 target/release/ai-firewall /usr/local/bin/ai-firewall

/metrics shows one in-flight request​

This is normal because the metrics request itself is active.

Wrong OpenAI-compatible base URL​

Use a provider root URL or /v1 base path.

Correct:

upstream_base_url http://ollama:11434/v1;

Wrong:

upstream_base_url http://ollama:11434/v1/chat/completions;

Upstream provider errors​

Check:

aif_errors_total{class="upstream_authentication_error"}
aif_errors_total{class="upstream_not_found"}
aif_errors_total{class="upstream_rate_limited"}
aif_errors_total{class="upstream_tls_error"}
aif_errors_total{class="upstream_dns_error"}
aif_errors_total{class="upstream_connect_error"}

Common causes:

wrong provider API key

  • full endpoint path configured instead of base URL
  • provider hostname cannot be resolved
  • provider port is unreachable
  • self-signed or hostname-mismatched TLS certificate

Embedding provider timeouts​

Check:

aif_embedding_timeouts_total
aif_embedding_request_duration_seconds

Common causes:

  • wrong provider API key
  • full endpoint path configured instead of base URL
  • provider hostname cannot be resolved
  • provider port is unreachable
  • self-signed or hostname-mismatched TLS certificate

Dashboards are empty​

Common causes:

  • no traffic has been sent yet
  • Prometheus is not scraping the firewall
  • Grafana datasource is not connected
  • wrong Compose working directory
  • dashboard provisioning paths are wrong

Check:

curl http://localhost:8080/metrics

and open:

http://localhost:9090/targets

Security Guard blocks or errors​

Module overview: VCAL Security Guard.

Symptoms:

HTTP/1.1 403 Forbidden
security_request_blocked
security_response_blocked
security_guard_unavailable
security_guard_timeout

Common causes:

  • Security Guard detected prompt injection, jailbreak, or system-prompt extraction text;
  • Security Guard is running in enforce mode;
  • Security Guard is unavailable and guard_fail_open false;
  • API key mismatch between AI Firewall and Security Guard;
  • wrong security_guard_url;
  • Security Guard default mode is still detect_only when blocking was expected.

Recommended checks:

curl http://localhost:8091/healthz
curl http://localhost:8091/readyz
curl -s http://localhost:8080/metrics | grep -E 'aif_guard_requests_total|aif_security_blocks_total'

Check configuration:

security_guard_enabled true;
security_guard_url http://vcal-security-guard:8091;
security_guard_api_key dev-security-key;
security_guard_timeout_seconds 3;
guard_fail_open false;

For production-like enforcement tests, Security Guard should normally use:

VCAL_SECURITY_GUARD_DEFAULT_MODE=enforce

Privacy Guard restore or anonymization errors​

Module overview: VCAL Privacy Guard.

Symptoms:

privacy_guard_unavailable
privacy_guard_timeout
privacy_restore_failed
guard_contract_violation

or final responses contain placeholders such as [EMAIL_1] or [IP_1].

Common causes:

  • Privacy Guard unavailable and guard_fail_open false;
  • API key mismatch between AI Firewall and Privacy Guard;
  • wrong privacy_guard_url;
  • expired or missing mapping ID;
  • restore disabled;
  • response was blocked by Security Guard before restore;
  • non-text content was expected to be scanned or restored.

Recommended checks:

curl http://localhost:8090/healthz
curl http://localhost:8090/readyz
curl -s http://localhost:8090/metrics | grep vcal_privacy
curl -s http://localhost:8080/metrics | grep -E 'aif_guard_requests_total|aif_privacy_restore_skipped_total'

VCAL Audit delivery failures​

Module overview: VCAL Audit.

Symptoms include missing traces in Audit, delayed evidence delivery, repeated delivery retries, or dropped batches in AI Firewall logs.

Recommended checks:

curl http://localhost:8092/healthz
curl http://localhost:8092/readyz
curl http://localhost:8092/metrics
curl -s -H "Authorization: Bearer $AUDIT_API_KEY" \
  "http://localhost:8092/v1/events?limit=1&after_sequence=0"

Verify:

  • audit_enabled true;
  • audit_url resolves from the AI Firewall container;
  • AI Firewall and VCAL Audit share the expected Docker network;
  • the configured Audit API key matches the service;
  • VCAL Audit accepts POST /v1/events/batch;
  • AI Firewall logs show buffered Audit delivery initialization.

The sender queue is memory-backed. A batch that remains undeliverable after retry exhaustion can be dropped and is not replayed automatically after restart.

VCAL Compliance cannot read Audit evidence​

Module overview: VCAL Compliance.

VCAL Compliance runs downstream of VCAL Audit. If Compliance is healthy but no evaluations or evidence-backed results appear, verify the Audit side first.

Recommended checks:

curl http://localhost:8093/healthz
curl http://localhost:8093/readyz
curl http://localhost:8093/metrics
curl http://localhost:8092/healthz
curl http://localhost:8092/readyz

Common causes:

  • VCAL Audit is unavailable or not ready;
  • Compliance is configured with the wrong Audit URL;
  • vcal-compliance cannot resolve or reach vcal-audit:8092 on the Docker network;
  • API credentials between Compliance and Audit do not match;
  • Audit has not received the expected AI Firewall evidence yet;
  • the Compliance control configuration is missing or invalid.

From the shared Docker network, both services should be reachable at their service names:

vcal-audit:8092
vcal-compliance:8093

Troubleshoot the chain in order: AI Firewall -> VCAL Audit -> VCAL Compliance.

Streaming requests rejected​

AI Firewall v0.4.2 rejects requests with:

{
  "stream": true
}

with HTTP 422 before cache, guard, or upstream processing.

Use non-streaming requests in every deployment mode.

Non-text content is not scanned​

The current guard modules inspect text content only. Images, audio, video, and binary payloads may be preserved where possible but are not scanned, anonymized, or classified by AI Firewall guard modules.

If a chatbot extracts OCR text, captions, or transcripts and sends them as text, that extracted text can be scanned and anonymized normally.

Missing or duplicate terminal evidence events​

Inspect request lifecycle events:

docker compose logs firewall 2>&1   | grep -E 'request\.(received|completed|failed)'

Every trace that emits request.received must have exactly one terminal event: either request.completed or request.failed.

Enable evidence logs with:

RUST_LOG=info,vcal_evidence=info

If lifecycle events are missing, confirm that the running image is v0.4.2 and that vcal_evidence=info is not filtered by the logging configuration.

VCAL Audit delivery failures​

Check AI Firewall logs for buffered-delivery initialization, retries, and dropped batches.

Verify:

  • audit_enabled true;
  • audit_url resolves from the AI Firewall container
  • AI Firewall and VCAL Audit share a Docker network
  • the API key matches VCAL_AUDIT_API_KEY
  • VCAL Audit /readyz returns HTTP 200
  • VCAL Audit accepts POST /v1/events/batch

The sender queue is memory-backed. A batch is dropped after retry exhaustion and is not replayed automatically after restart.