Os Labs / AI Engineering
AI EngineeringWorking~5h
Build a semantic cache for LLM calls and prove the cost, latency, and correctness tradeoffs
A caching layer that sits in front of an LLM API and serves cached responses for semantically equivalent requests (paraphrases, reordered words, minor typos) using embedding similarity against a similarity threshold, not just exact-string keys. You instrument it to track hit rate, dollars saved, and latency saved across a realistic query mix, and you implement explicit invalidation (TTL plus manual purge) so the cache doesn't silently serve outdated answers after the underlying prompt, model, or system context changes.
O que isso prova
A frase que você pode defender em uma entrevista.
Linha do currículo
Built a semantic LLM response cache with embedding-similarity keying and TTL/manual invalidation, measuring a 40%+ cache hit rate and cost/latency reduction across a 100-query mixed workload while catching over-eager false-positive cache hits.
- Understands why exact-string caching misses most real-world duplicate traffic and how semantic keying closes that gap
- Can pick and justify a similarity threshold that balances hit rate against false-positive risk
- Knows how to instrument a system to prove cost and latency claims with real numbers, not estimates
- Thinks about cache invalidation and staleness as a first-class correctness problem, not an afterthought
O briefing
O que você constrói, passo a passo.
- 01Implement a cache lookup that embeds incoming requests and checks for a prior request within a configurable cosine-similarity threshold, returning the cached response on a hit
- 02Implement cache writes that store the response keyed by the request embedding plus a TTL and a version tag tied to the prompt template/model version in use
- 03Build a 100+ query test workload that mixes exact repeats, paraphrases, near-duplicates, and genuinely distinct queries, and run it through both a no-cache baseline and the semantic cache
- 04Instrument and report cache hit rate, total LLM API cost, and p50/p95 latency for both configurations, showing the measured delta
- 05Implement explicit invalidation: a TTL expiry path and a manual purge-by-version path triggered when the prompt template or model changes, and prove a stale entry is never served after either
- 06Stress-test the similarity threshold with at least 5 near-miss queries designed to be similar-but-not-equivalent (e.g. negated or entity-swapped versions of a cached query) and document any false-positive hits and how the threshold was tuned in response
A prova
Está pronto quando estes passarem.
Cache hit rate, total cost, and p50/p95 latency are measured and reported for cached vs uncached runs over the same 100+ query workload
automated test
After a TTL expiry or a version-tag change, a request that would have hit the stale cache entry instead misses and triggers a fresh LLM call
automated test
At least one near-miss query in the false-positive stress test is caught and the similarity threshold or key design is adjusted with the reasoning documented
automated test
A report shows the cost/latency comparison table plus the invalidation test results
artifact
Stack
Pythonan LLM APIan embedding model for semantic keysa key-value store (Redis or SQLite) with TTL support
Método Sage
frame → route → decide → prove
Você mantém
A semantic LLM cache with TTL/version invalidation + cost/latency comparison report + false-positive stress-test log