AI Reliability Audit
A representative Sage Audit, with client identifiers redacted. This is the exact shape of what you receive: a scorecard, severity-ranked findings with evidence, a risk register, and a prioritized four-week remediation plan you could hand to any engineer.
● Client name, repository, and figures redacted / illustrative.
1 · Scorecard
| Dimension | Grade | One-line read |
|---|---|---|
| Reliability & evals | C+ | No regression suite on the RAG pipeline; quality measured by vibe. |
| Security & secrets | B- | Keys in env, but one provider key reachable from the client bundle. |
| Cost & latency | B | p95 latency 4.1s; ~40% spend on an oversized model for classify steps. |
| Data & retrieval | C | Chunking is naive; retrieval precision unmeasured; no citations stored. |
| Observability | D+ | No tracing on LLM calls; failures are invisible until a user reports them. |
| Architecture & scale | B+ | Clean service boundaries; a single synchronous path will cap throughput. |
2 · Findings
- critical
Provider API key reachable in the client bundle
The model provider key is referenced through a NEXT_PUBLIC_* variable and ships in the browser bundle, allowing anyone to spend against the account. Rotate immediately and proxy all model calls server-side.
- critical
No evaluation harness — quality is unmeasured
There is no offline eval set or scoring. Every prompt or model change is a blind deploy. A 40-case golden set with automated scoring would catch regressions before users do.
- high
Retrieval precision is unknown and likely low
Fixed 1,000-char chunking splits tables and code mid-structure. No retrieval metric is captured. Estimated 1 in 3 answers cites the wrong passage. Recommend semantic chunking + a retrieval@k measurement.
- high
Oversized model on cheap steps
The flagship model runs classification and routing that a small model handles at equal accuracy — ~40% of spend and ~1.2s of p95 latency for no quality gain.
- medium
No tracing on LLM calls
Failures, timeouts, and bad outputs are invisible. A single tracing layer (inputs, outputs, latency, cost per call) turns "a user complained" into a dashboard.
- medium
Synchronous single-path inference caps throughput
All requests share one synchronous path with no queue or backpressure. Under load, p95 degrades non-linearly. A job queue with graceful shutdown decouples spikes.
3 · Remediation plan
| Week 1 | Rotate the exposed key; move all model calls server-side behind a proxy. Stand up basic tracing. |
| Week 2 | Build a 40-case golden eval set + automated scoring wired into CI. Establish the baseline. |
| Week 3 | Swap the small-model classify/route steps; re-run evals to prove no quality loss + capture the cost delta. |
| Week 4 | Semantic chunking + retrieval@k measurement + stored citations. Re-score against the golden set. |
Every finding ships with the evidence behind it and a fix specific enough to act on this week — not a slide deck of generalities. That is the deliverable.