Labs / AI Engineering
AI EngineeringWorking~6h
Build a semantic search index and measure retrieval quality
A semantic search engine over a real dataset of your choosing (e.g. a docs corpus, a set of support tickets, a product catalog, your own notes — at least 500 documents/chunks). You build the embedding pipeline, chunking strategy, and vector index, implement ranked retrieval with a similarity metric, and then rigorously evaluate it: you hand-label a query/relevant-document ground-truth set and measure recall@k and precision@k against it, comparing your semantic approach to a plain keyword-search baseline on the same queries.
Что это доказывает
Строка, которую вы сможете защитить на собеседовании.
Строка для резюме
Built a semantic search index over 500+ real documents (chunking, embedding, vector index, ranked retrieval) and evaluated it against a hand-labeled ground-truth set, measuring recall@k/precision@k and benchmarking against a keyword-search baseline.
- Can build a real embeddings pipeline end to end, not just call an API once
- Understands chunking strategy and why it materially affects retrieval quality
- Knows how to build ground truth and measure retrieval with recall@k/precision@k instead of eyeballing results
- Can honestly compare an AI approach against a simpler baseline instead of assuming it wins
Краткое описание
Что вы создаёте, шаг за шагом.
- 01Choose a real dataset of >=500 documents or chunks (public docs, your own writing, an open dataset — must be real content, not synthetic filler) and define a chunking strategy (size, overlap, boundary rules) with a stated rationale
- 02Generate embeddings for every chunk and build a vector index supporting top-k nearest-neighbor retrieval by cosine similarity (or your chosen metric)
- 03Implement a keyword-search baseline (BM25 or TF-IDF) over the same chunked corpus using the same chunks, so the comparison is apples-to-apples
- 04Hand-write >=20 realistic queries against the dataset and manually label the truly relevant chunk(s) for each — this ground-truth set must be built before running either retrieval method against it
- 05Run both the semantic index and the keyword baseline over all 20+ queries, computing recall@5 and precision@5 (or your chosen k) for each system
- 06Write a results report comparing the two systems, including at least 2 concrete cases where semantic search wins (paraphrase/synonym queries) and at least 1 case where keyword search wins or ties (exact term/code/ID lookups), with an honest explanation of why
Доказательство
Готово, когда эти проверки пройдены.
The vector index returns nearest-neighbor results for a query in under 1 second for the full corpus size used
automated test
Recall@5 and precision@5 are computed against the hand-labeled ground-truth set for both the semantic index and the keyword baseline, not just one
automated test
At least one documented query shows semantic search retrieving a relevant chunk that the keyword baseline misses due to paraphrasing/synonymy
automated test
A results report exists with the recall@k/precision@k table for both systems and a written comparison
artifact
Стек
Pythonan embeddings API or local embedding modela vector index (FAISS, Chroma, or pgvector)a keyword search baseline (BM25 or simple TF-IDF)
Метод Sage
frame → map → decide → prove
Вы сохраняете
A semantic search index + keyword-search baseline + labeled ground truth + retrieval-quality comparison report