End-to-End Big Data Pipeline

A complete, production-ready data engineering project demonstrating a modern data pipeline architecture using containerized services.

This project implements a comprehensive data pipeline that ingests real-time data, processes it at scale, stores it in multiple databases, and visualizes insights through interactive dashboards.

Data Flow:

Random User API → Airflow → Kafka → Spark → Cassandra (Data) + PostgreSQL (Metadata) → Superset (Visualization)

Key Insight: PostgreSQL is used exclusively for metadata (Airflow DAGs, Superset dashboards). Application data flows through Kafka → Spark → Cassandra. Redis provides in-memory caching for performance.

See ARCHITECTURE.md for detailed component descriptions and deployment information.

• Docker & Docker Compose
• Python 3.10+
• Git

1. Clone the repository

git clone <repository-url>
cd E2E

2. Start all services

docker compose up -d

3. Verify services are running

docker compose ps

4. Access the services

• Airflow UI: http://localhost:8080 (metadata orchestration)
• Superset UI: http://localhost:8088 (dashboarding)
• Kafka: localhost:9092 (message broker)
• Spark Master UI: http://localhost:8080 (processing)
• PostgreSQL: localhost:5432 (metadata only)
• Cassandra: localhost:9042 (application data)
• Redis: localhost:6379 (caching)

E2E/
├── airflow/                      # Airflow orchestration
│   ├── dags/
│   │   └── kafka_stream.py      # Main streaming DAG
│   ├── docker-compose.yml        # Airflow services
│   ├── Dockerfile               # Custom Airflow image
│   ├── requirements.txt          # Python dependencies
│   └── config/
│       └── airflow.cfg          # Airflow configuration
├── kafka/                        # Kafka setup
│   └── docker-compose.yml        # Kafka & Zookeeper services
├── spark/                        # Spark processing
│   └── docker-compose.yml        # Spark services
├── postgres/                     # PostgreSQL database
│   ├── docker-compose.yml
│   ├── airflow_init.sql         # Airflow DB initialization
│   └── superset_init.sql        # Superset DB initialization
├── cassandra/                    # Cassandra NoSQL database
│   └── docker-compose.yml
├── superset/                     # Superset BI platform
│   ├── docker-compose.yml
│   └── docker/
│       ├── superset_config.py    # Superset configuration
│       ├── docker-bootstrap.sh   # Bootstrap script
│       └── .env                  # Environment variables
├── docker-compose.yml            # Root compose (includes all services)
└── scripts/                      # Utility scripts
    └── entrypoint.sh            # Service initialization

The DAG in airflow/dags/kafka_stream.py performs:

• Fetches random user data from RandomUser.me API
• Formats and structures the data
• Produces messages to Kafka topic users_created
• Scheduled to run periodically

Topics:

• users_created - Raw user data stream from API

Spark Structured Streaming jobs consume from Kafka and:

• Read from users_created Kafka topic
• Apply transformations and data quality checks
• Enrich data with additional attributes
• Write processed data to Cassandra (primary storage)

PostgreSQL (Metadata Only):

• Airflow DAG definitions and task state
• Superset dashboard/slice definitions and user sessions
• DO NOT use for application data

Redis (In-Memory Cache):

• Caches Airflow task state and DAG metadata
• Caches Superset query results for fast dashboard loads
• Stores session information for both Airflow and Superset

Cassandra (NoSQL - Primary Storage):

• User profiles and events
• Time-series data from Spark processing
• High write-throughput, distributed architecture
• Replicated across cluster for durability

Interactive dashboards displaying:

• User demographics (from Cassandra)
• Activity patterns and trends
• Real-time metrics and KPIs
• Metadata and dashboard state (from PostgreSQL)
• Cached query results (from Redis)

Key environment files:

• postgres/.env - PostgreSQL configuration
• superset/docker/.env - Superset configuration

Modify these to customize:

• Database credentials
• Service ports
• Log levels
• Performance tuning parameters

Edit airflow/config/airflow.cfg for:

• Parallelism settings
• Executor configuration
• DAG parsing behavior

CREATE TABLE users (
    user_id SERIAL PRIMARY KEY,
    first_name VARCHAR(255),
    last_name VARCHAR(255),
    email VARCHAR(255) UNIQUE,
    phone VARCHAR(20),
    nationality VARCHAR(5),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE user_events (
    event_id UUID PRIMARY KEY,
    user_id INT,
    event_type TEXT,
    event_data MAP<TEXT, TEXT>,
    timestamp BIGINT,
    created_at TIMESTAMP
);

# Airflow logs
docker logs airflow-scheduler

# Spark logs
docker logs spark-master

# Kafka logs
docker logs broker

# PostgreSQL logs
docker logs postgres

# Check all services
docker compose ps

# Check Kafka broker
docker exec broker kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# Check PostgreSQL connection
psql -h localhost -U airflow -d airflow

Database Connection Error

# Verify network connectivity
docker network inspect e2e_airflow-network

# Restart database
docker compose -f postgres/docker-compose.yml restart

Kafka Producer Issues

# Check Kafka broker status
docker logs broker | grep ERROR

# Verify topic exists
docker exec broker kafka-topics.sh --list --bootstrap-server localhost:9092

1. Security

   • Use environment-specific .env files
   • Enable SSL/TLS for database connections
   • Implement authentication for all services

2. Scaling

   • Increase Spark executor count
   • Configure Cassandra replication factor
   • Set up Kafka partitioning strategy

3. Monitoring

   • Implement centralized logging (ELK stack)
   • Add metrics collection (Prometheus)
   • Set up alerting mechanisms

4. Backup & Recovery

   • PostgreSQL backup strategy
   • Cassandra snapshot management
   • Kafka topic retention policies

• Apache Airflow - Workflow orchestration (metadata in PostgreSQL)
• Apache Kafka - Event streaming platform
• Apache Spark - Distributed data processing
• PostgreSQL - Relational database (metadata only, NOT for OLTP)
• Apache Cassandra - NoSQL database (primary application data storage)
• Redis - In-memory cache (session & query result caching)
• Apache Superset - Data visualization & BI
• Docker & Docker Compose - Containerization

1. Create a feature branch (git checkout -b feature/improvement)
2. Make your changes
3. Commit with descriptive messages
4. Push to the branch
5. Open a Pull Request

This project is open source and available under the MIT License.

For issues, questions, or suggestions:

• Open an issue on GitHub
• Create a discussion for feature requests

Built with ❤️ for learning and production-ready data engineering