You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
End-to-end Metadata-Driven Data Engineering framework built on Azure. Features dynamic SQL/REST API ingestion with range pagination, automated schema mapping, and event-driven orchestration. Implements robust CI/CD via GitHub Actions/YAML and automated failure alerting with Logic Apps. Optimized for scalability and DE best practices.
Metadata-driven incremental ingestion from Azure SQL to ADLS Gen2 with ADF (CDC watermarks, alerts) feeding a Databricks Delta Lake medallion lakehouse.
This project implements a comprehensive event-driven data pipeline for e-commerce transactional data processing using Databricks, PySpark, and Delta Lake. The pipeline handles multiple data sources with advanced data engineering patterns including SCD2 (Slowly Changing Dimensions), data validation, enrichment, and automated archiving.
Production-style data engineering portfolio project for a Legal CRM: API ingestion, Airflow orchestration, Polars transformations, SQL Server medallion architecture, incremental loads, JSON logs and automated tests.
Incremental data ingestion pipeline that extracts GitHub API data for the Spotify organization and loads it into BigQuery using Python, Airflow, and Docker.
Incremental stock market data pipeline on Databricks: Alpha Vantage API → Auto Loader → Delta Lake bronze/silver/gold, with SCD Type 2 company history via MERGE.
Incremental ETL pipeline for the French power mix (RTE éCO2mix, 30-min since 2012) into a partitioned PostgreSQL warehouse: watermark + lookback, version priority, revision history, UTC & DST handling, 41 tests, bilingual LaTeX report.
Incremental ETL in Python: watermark windows that cannot silently skip rows, idempotent upserts, and validation that quarantines bad records instead of failing the batch. 48 tests
🚲 Pipeline Microsoft Fabric : les vélos BIXI de Montréal. Un Copy job incrémentiel qui retient son filigrane, une copie typée, une inbox consommée, l'effet de la pluie mesuré en SQL et publié dans Teams.
Incremental ELT from a paginated ERP API into Snowflake: 24 entities from one declarative catalog, merged by row hash so only real changes are written. Synthetic data.
Incremental ELT pipeline over the public PyPI, npm and crates.io APIs into a DuckDB warehouse, with watermarks, conditional requests, backfills and a quality gate