Os Labs / AI Engineering
AI EngineeringAdvanced~6h
Add a cross-encoder re-ranker to a RAG pipeline and prove it improves recall@k
A two-stage retrieval pipeline over a real document corpus: stage one does wide bi-encoder retrieval (top 50-100 candidates via cosine similarity on embeddings), stage two re-scores those candidates with a cross-encoder that jointly encodes query and passage, then returns a re-ordered top-k. You build a labeled eval set of query-to-relevant-passage pairs and compute recall@k and MRR before and after re-ranking, so the improvement is a number, not a vibe.
O que isso prova
A frase que você pode defender em uma entrevista.
Linha do currículo
Built a two-stage RAG retrieval pipeline (bi-encoder recall + cross-encoder re-ranking) and measured a recall@5 and MRR improvement against a labeled eval set of 40+ query-passage pairs.
- Understands why bi-encoder similarity alone under-ranks relevant passages, and what a cross-encoder buys you
- Can build and defend a labeled retrieval eval set instead of asserting quality by inspection
- Knows the latency/cost tradeoff of re-ranking and where to draw the candidate-pool cutoff
- Can quantify a retrieval improvement with standard IR metrics (recall@k, MRR)
O briefing
O que você constrói, passo a passo.
- 01Ingest a real multi-document corpus (100+ chunks minimum) and build a bi-encoder embedding index for stage-one retrieval
- 02Hand-label a query set of 40+ queries, each with at least one known-relevant chunk ID, drawn from the actual corpus (no synthetic gold labels copy-pasted from the passage text)
- 03Implement stage-one retrieval that returns the top 50-100 candidates by embedding similarity
- 04Implement stage-two re-ranking with a cross-encoder that scores each (query, candidate) pair independently and re-sorts to a final top-k
- 05Compute recall@k (k=3, k=5) and MRR for bi-encoder-only vs bi-encoder+re-ranker on the full labeled query set, and report the delta
- 06Measure and report added latency per query from the re-ranking stage, and state the candidate-pool size you chose as the latency/quality tradeoff point
A prova
Está pronto quando estes passarem.
recall@5 and MRR are computed against the labeled eval set for both bi-encoder-only and bi-encoder+re-ranker configurations
automated test
The re-ranked configuration shows a measured recall@5 or MRR improvement over bi-encoder-only on the labeled set, with the numbers reported not asserted
automated test
For at least 3 example queries, a passage that ranked outside top-5 in stage one is promoted into the final top-5 after re-ranking, and this is shown with before/after rankings
automated test
A report captures the eval methodology, the labeled query set, per-config metrics, and measured added latency from re-ranking
artifact
Stack
Pythona bi-encoder embedding modela cross-encoder scoring model (e.g. an MS MARCO-trained cross-encoder)a vector store or in-memory cosine searchan eval harness for recall@k / MRR
Método Sage
frame → route → prove
Você mantém
A two-stage retrieval pipeline + labeled eval set + before/after recall@k and MRR report