Los Laboratorios / AI Engineering
AI EngineeringWorking~5h
Build a semantic cache for LLM calls and prove the cost, latency, and correctness tradeoffs
A caching layer that sits in front of an LLM API and serves cached responses for semantically equivalent requests (paraphrases, reordered words, minor typos) using embedding similarity against a similarity threshold, not just exact-string keys. You instrument it to track hit rate, dollars saved, and latency saved across a realistic query mix, and you implement explicit invalidation (TTL plus manual purge) so the cache doesn't silently serve outdated answers after the underlying prompt, model, or system context changes.
Lo que demuestra
La frase que puedes defender en una entrevista.
Línea de currículum
Built a semantic LLM response cache with embedding-similarity keying and TTL/manual invalidation, measuring a 40%+ cache hit rate and cost/latency reduction across a 100-query mixed workload while catching over-eager false-positive cache hits.
- Understands why exact-string caching misses most real-world duplicate traffic and how semantic keying closes that gap
- Can pick and justify a similarity threshold that balances hit rate against false-positive risk
- Knows how to instrument a system to prove cost and latency claims with real numbers, not estimates
- Thinks about cache invalidation and staleness as a first-class correctness problem, not an afterthought
El resumen
Lo que construyes, paso a paso.
- 01Implement a cache lookup that embeds incoming requests and checks for a prior request within a configurable cosine-similarity threshold, returning the cached response on a hit
- 02Implement cache writes that store the response keyed by the request embedding plus a TTL and a version tag tied to the prompt template/model version in use
- 03Build a 100+ query test workload that mixes exact repeats, paraphrases, near-duplicates, and genuinely distinct queries, and run it through both a no-cache baseline and the semantic cache
- 04Instrument and report cache hit rate, total LLM API cost, and p50/p95 latency for both configurations, showing the measured delta
- 05Implement explicit invalidation: a TTL expiry path and a manual purge-by-version path triggered when the prompt template or model changes, and prove a stale entry is never served after either
- 06Stress-test the similarity threshold with at least 5 near-miss queries designed to be similar-but-not-equivalent (e.g. negated or entity-swapped versions of a cached query) and document any false-positive hits and how the threshold was tuned in response
La prueba
Está terminado cuando estos pasan.
Cache hit rate, total cost, and p50/p95 latency are measured and reported for cached vs uncached runs over the same 100+ query workload
automated test
After a TTL expiry or a version-tag change, a request that would have hit the stale cache entry instead misses and triggers a fresh LLM call
automated test
At least one near-miss query in the false-positive stress test is caught and the similarity threshold or key design is adjusted with the reasoning documented
automated test
A report shows the cost/latency comparison table plus the invalidation test results
artifact
Stack
Pythonan LLM APIan embedding model for semantic keysa key-value store (Redis or SQLite) with TTL support
Método Sage
frame → route → decide → prove
Conservas
A semantic LLM cache with TTL/version invalidation + cost/latency comparison report + false-positive stress-test log