Os Labs / AI Engineering
AI EngineeringWorking~5h
Build a structured-output extractor with schema-conformance guarantees
An extraction pipeline that takes messy real-world text (e.g. job postings, invoices, support tickets, resumes — pick one real domain) and reliably returns data conforming to a strict JSON schema, even when the source text is incomplete, ambiguous, or adversarially malformed. You implement schema-constrained generation, validation against the schema, and an automatic repair loop for validation failures, then prove conformance across a stress-test suite including edge cases designed to break naive parsing.
O que isso prova
A frase que você pode defender em uma entrevista.
Linha do currículo
Built a structured-output extraction pipeline (schema-constrained generation + JSON Schema validation + automatic repair loop) achieving 100% schema conformance across a 30+ case stress-test suite including malformed and adversarial inputs.
- Can go beyond 'ask the model for JSON and hope' to actually guaranteeing conformance
- Understands JSON Schema validation and how to use it as a hard gate, not a suggestion
- Can design a repair/retry loop that fixes failures instead of just erroring out
- Tests against edge cases and adversarial inputs, not just the happy path
O briefing
O que você constrói, passo a passo.
- 01Pick one real extraction domain (e.g. job posting -> {title, company, salary_range, remote, required_skills[]}) and write a strict schema (required fields, types, enums, nested arrays) using JSON Schema or a Pydantic/Zod model
- 02Implement the extractor using the model's native structured-output or tool-calling mode (not just a 'return JSON' instruction in the prompt)
- 03Validate every output against the schema programmatically; on validation failure, feed the specific validation error back to the model and retry with a bounded attempt limit (e.g. max 3 repair attempts) rather than retrying blind
- 04Build a stress-test suite of >=30 inputs: clean cases, missing-field cases, ambiguous cases (data present but unclear), and adversarial cases (text that tries to inject fake JSON or break the parser)
- 05Run the full suite and report the conformance rate, the repair-loop success rate (how many failures were fixed by retry vs. exhausted attempts), and which input patterns caused the most failures
- 06Handle the unrecoverable case explicitly: after exhausting repair attempts, return a typed error/failure result instead of silently returning invalid or null data
A prova
Está pronto quando estes passarem.
Every output that is returned as 'success' validates against the JSON Schema with zero exceptions across all 30+ stress-test cases
automated test
At least one deliberately malformed/ambiguous input triggers the repair loop and is successfully corrected within the attempt limit
automated test
An input engineered to be unrecoverable (e.g. required data genuinely absent from the text) returns an explicit typed failure, not a silently invalid or fabricated JSON object
automated test
A conformance report exists showing pass rate, repair success rate, and a breakdown of failure patterns by input category
artifact
Stack
Pythonan LLM API with structured-output/tool-calling supporta JSON Schema validator (e.g. jsonschema/Pydantic/Zod)
Método Sage
frame → decide → prove
Você mantém
A structured-output extractor + JSON Schema + stress-test suite + conformance report