What
This is an ELT pipeline that loads SpaceX launch data from the REST API into Snowflake. We do this with Singer (Python) and dbt.
Why
I wanted to practise writing modular ELT pipelines and using cloud data warehouses. The goal was to simulate a production data ingestion system.
How
I wrote a custom Singer tap in Python and used dbt to transform the data after it landed in Snowflake.
Learnings
I learned how to handle rate limits, write idempotent tasks, and deal with flaky upstream data. Biggest lesson: write small, testable components early.
https://github.com/ivanovyordan/course-data-engineering-spacex