dbt-Airflow-POC

This repository provides a proof-of-concept (POC) for using dbt (data build tool) and Airflow to manage data transformations across two PostgreSQL environments (dev and staging) using CI/CD practices with GitHub Actions and Airflow orchestration.

• Prerequisites
• Setup
  • Clone the Repository
  • Build and Run the Containers
• Running dbt Commands
• Directory Structure
• Airflow DAGs
• License

• Docker
• Docker Compose

git clone https://github.com/noyb34/dbt-poc.git
cd dbt-poc

docker-compose up -d --build

You can run dbt commands directly using Docker Compose. Make sure to navigate to the project root directory.

1. Run dbt Seed to Load Sample Data:

   docker-compose run dbt dbt seed

2. Run dbt Models to Perform Transformations:

   docker-compose run dbt dbt run 

3. Run dbt Tests to Validate Transformations:

   docker-compose run dbt dbt test

project-root/
├── config/
├── dags/
├── dbt_project/
│   ├── dbt_packages/
│   ├── logs/
│   ├── models/
│   ├── seeds/
│   ├── target/
│   ├── tests/
│   ├── .user.yml
│   ├── dbt_project.yml
│   └── profiles.yml
├── logs/
├── pg_config/
├── plugins/
├── .gitignore
├── docker-compose.yml
├── Dockerfile_dbt
├── Dockerfile_postgres
├── example.env
└── LICENSE

Place your Airflow DAGs in the dags/ directory. Airflow will automatically pick up and schedule these DAGs.

This project is licensed under the GPL-3.0 License. See the LICENSE file for details.