say hello

Logistics Customer Ops Pipeline

Customer data pipeline · Feb–Jun 2026 · Public repo

A data engineering pipeline that turns messy logistics exports into trusted operations tables, reject reports, SLA views, exception reports, route productivity outputs, and a dashboard.

GitHub repo All work

What it is

The product story is a small logistics control tower for operators dealing with messy third-party exports.

The dashboard and CSV reports read built artifacts rather than hand-coded marketing numbers, so the demo ties back to ingested files, reject evidence, quality checks, and marts.

The loop it has to make believable

  1. 01

    Built config-driven ingestion that maps messy customer exports into canonical Bronze parquet.

  2. 02

    Added JSON Schema validation, reject reports for bad rows, and an ingestion ledger so reruns skip identical files.

How it’s built

The verified path is intentionally pragmatic: customer CSVs, config-driven transforms, Bronze parquet, DuckDB source views, dbt models, quality checks, CSV exports, and Streamlit.

The deeper path shows how the same domain could expand into Kafka topics, Spark processing, and Airflow orchestration, but the local customer demo is the source of truth.

Architecture

Ingestion
Customer CSV exports -> config-driven mapping -> canonical Bronze parquet with contracts, rejects, and idempotency
Modeling
DuckDB source views plus dbt staging, intermediate, and mart layers
Outputs
Shipment summary, SLA breaches, delivery exceptions, trip summary, route productivity, and Streamlit dashboard
Depth path
Kafka, Spark, and Airflow exist as optional depth, not the main proof path

What has to hold up

  • Modeled shipments, SLA breaches, delivery exceptions, trips, and route productivity in DuckDB/dbt.
  • Kept Kafka, Spark, and Airflow as optional architecture depth rather than leading with unverified streaming claims.

How I would present it

I would start with the dashboard and then trace one metric backward through dbt and source data.

The honest boundary is that the local customer batch path is the strongest proof; the streaming stack is architecture depth unless freshly run end to end.

Proof

The strongest proof is the local customer batch path: `make customer-trial` plus verification that the core marts and CSV exports are populated.

VerifyRepository, checks, stack, status
  1. 01

    Open the GitHub repo.

  2. 02

    Read the architecture and edge-case sections.

  3. 03

    Follow the documented proof path: README, make customer-trial, make customer-verify, dbt project, quality reports, Streamlit app.

Role
Config-driven ingestion, validation, dbt marts, quality checks, exports, dashboard
Scope
Customer CSV exports to trusted logistics marts, reports, and dashboard
Proof
README, make customer-trial, make customer-verify, dbt project, quality reports, Streamlit app
Stack
Python, DuckDB, dbt, Streamlit, Kafka, Spark, Airflow
Core marts
5 non-empty
Exports
5 CSVs
Warehouse
DuckDB/dbt
Proof
make customer-trial