AI and AutomationStatus: Live

Marketplace Data Engineering Pipeline

Local, Reproducible Pipeline with Automated Matching & Resumable Repairs

[MARKETPLACE PIPELINE ARCHITECTURE DIAGRAM]

Conceptual architecture and public-safe interface diagram

Harrison's Contribution

Sole Developer and Architect. Designed the database schema, wrote the pipeline execution and repair modules, authored the test suite, and documented the codebase.

Intended Stakeholders

Data engineers, analytics engineering teams, and technical hiring managers seeking evidence of rigorous data modeling, testing discipline, and resilient architectural design.

Core Technologies

Python 3.11+DuckDBSQL

Project Overview

A fully reproducible, production-grade local data pipeline engineered in Python 3.11+ using DuckDB. Developed to model a high-throughput fictional marketplace, this project showcases clean data engineering, fuzzy product matching, safe stateful repairs, and indexed daily statistics without requiring cloud infrastructure credentials.

The Problem & Opportunity

Traditional enterprise data pipeline assessments and sandbox environments often rely on complex, expensive cloud infrastructure (AWS/GCP/Snowflake) that makes rapid local testing, deterministic verification, and peer review cumbersome and error-prone.

Key Objectives

Engineer a 100% self-contained local pipeline runnable with standard Python 3.11+ and DuckDB.

Generate synthetic noisy marketplace datasets with realistic corruption, duplicates, and edge cases.

Process an initial full snapshot followed by incremental daily batches.

Implement multi-stage product matching (categorized as matched, review, or rejected).

Support safe, idempotent, resumable data repair procedures and daily summary aggregations.

Solution & Implementation

Zero-Cloud-Dependency Architecture: Powered entirely by DuckDB for lightning-fast columnar SQL queries on local disk.

Deterministic Data Generation: Synthetic data generator with reproducible seed control for reproducible benchmarking.

Product Matching Engine: Multi-tier matching logic evaluating SKU similarity, title fuzzy matching, and category validation.

Stateful Resumable Repairs: CLI-driven repair tooling (`--wrong-id`, `--correct-id`, `--dry-run`) capable of safely modifying historical facts without corrupting primary key constraints.

Indexed Summary Statistics: High-speed reporting view enabling instant query response across arbitrary date ranges.

Technologies & Tooling

Python 3.11+DuckDBSQLPytestPowerShell AutomationData Modeling

Challenges & Key Decisions

Handling Concurrent Repair Edge Cases: Upserting repaired sales facts risked primary key collision in incremental tables. Decision: Implemented atomic merge and staging table deduplication to ensure idempotency.

Cross-Platform Testing: Ensuring consistent PowerShell behavior on Windows while maintaining full compatibility on Unix. Decision: Utilized pathlib and standardized CLI argument conventions.

Verified Results & Outcomes

Public repository actively hosted and verified at github.com/HarryBarri/marketplace-pipeline.
Comprehensive test coverage validating batch ingestion, edge-case recovery, and summary accuracy.
Proven architecture demonstrated to run seamlessly without cloud service fees or external network dependencies.

Project Evidence & Media

Public Source Code Repository

Complete Python and DuckDB source code, tests, and documentation.

Lessons Learned

Idempotency and dry-run safety are paramount in data repair systems.

Embedded columnar engines like DuckDB deliver astonishing analytics performance for testing and micro-services.

Future Opportunities

Adding a streaming ingestion wrapper with Kafka or Redis Streams.

Building a lightweight Streamlit dashboard for real-time monitoring of matching queues.

View Public GitHub Repository Collaborate on a Similar Project