ラボ / Data
DataWorking~8h
Data Quality Gate for Batch Ingestion
A pre-load validation gate that runs schema, null, range, and uniqueness checks against an incoming batch file before it's allowed into the target table. A single corrupt row fails the whole batch with a precise, row-level error report instead of silently loading partial or malformed data.
これが証明すること
面接で守れる一言。
履歴書の一言
Built a schema-and-constraint data quality gate for batch ingestion (type, null, range, and uniqueness checks) that blocks corrupt batches from loading and emits a row-level failure report, cutting bad-data incidents at the load boundary.
- decide
- prove
概要
ステップごとに作るもの。
- 01Ingestion pipeline reads a CSV batch and validates every row against a Pydantic schema (types, required fields) before any DB write is attempted
- 02Rule set includes: not-null checks on required columns, range checks (e.g. non-negative amounts, valid date bounds), and a uniqueness check on the batch's natural key
- 03A single failing row blocks the entire batch load (all-or-nothing) — no partial writes reach the target table
- 04Validation produces a structured report (row number, column, rule violated, offending value) written to a file, not just a log line
- 05Gate is runnable as a standalone check (dry-run mode) that validates without loading, for use in a pre-load CI step
- 06Passing batches are loaded inside a single transaction so the gate and the load share the same all-or-nothing guarantee
証明
これらが合格したら完了。
A batch with one row containing a null in a required column is fully rejected; zero rows land in the target table
automated test
A batch with a duplicate natural key across two rows is rejected with both offending row numbers reported
automated test
An out-of-range value (e.g. negative price) fails validation with the specific rule and value named in the report
automated test
A fully valid batch passes the gate and loads all rows in one transaction
automated test
技術スタック
PythonPandasPydanticPostgreSQLpytest
Sage Method
decide → prove
あなたが得るもの
CLI data-quality gate + sample corrupt/clean batch fixtures + row-level failure report + pytest suite