Os Labs / Data
DataWorking~8h
Data Quality Gate for Batch Ingestion
A pre-load validation gate that runs schema, null, range, and uniqueness checks against an incoming batch file before it's allowed into the target table. A single corrupt row fails the whole batch with a precise, row-level error report instead of silently loading partial or malformed data.
O que isso prova
A frase que você pode defender em uma entrevista.
Linha do currículo
Built a schema-and-constraint data quality gate for batch ingestion (type, null, range, and uniqueness checks) that blocks corrupt batches from loading and emits a row-level failure report, cutting bad-data incidents at the load boundary.
- decide
- prove
O briefing
O que você constrói, passo a passo.
- 01Ingestion pipeline reads a CSV batch and validates every row against a Pydantic schema (types, required fields) before any DB write is attempted
- 02Rule set includes: not-null checks on required columns, range checks (e.g. non-negative amounts, valid date bounds), and a uniqueness check on the batch's natural key
- 03A single failing row blocks the entire batch load (all-or-nothing) — no partial writes reach the target table
- 04Validation produces a structured report (row number, column, rule violated, offending value) written to a file, not just a log line
- 05Gate is runnable as a standalone check (dry-run mode) that validates without loading, for use in a pre-load CI step
- 06Passing batches are loaded inside a single transaction so the gate and the load share the same all-or-nothing guarantee
A prova
Está pronto quando estes passarem.
A batch with one row containing a null in a required column is fully rejected; zero rows land in the target table
automated test
A batch with a duplicate natural key across two rows is rejected with both offending row numbers reported
automated test
An out-of-range value (e.g. negative price) fails validation with the specific rule and value named in the report
automated test
A fully valid batch passes the gate and loads all rows in one transaction
automated test
Stack
PythonPandasPydanticPostgreSQLpytest
Método Sage
decide → prove
Você mantém
CLI data-quality gate + sample corrupt/clean batch fixtures + row-level failure report + pytest suite