Apache Spark on Polars

Blog about tech

The open source developer platform to build AI agents and models with confidence. Enhance your AI applications with end-to-end tracking, observability, and evaluations, all in one integrated platform.

Este repositorio contiene el material del curso de Udemy Big Data y Spark: ingeniería de datos con Python y pyspark. En este curso, aprenderás a utilizar las herramientas y técnicas necesarias para trabajar con grandes conjuntos de datos utilizando la librería pyspark.

A big data analysis of Los Angeles crime and demographic data using Apache Spark on Kubernetes.

This project is a centralized,fault-tolerant Data Engineering Pipeline designed to ingest,process & visualize user data generated in real-time.It leverages Apache Airflow for orchestration, Kafka for message buffering, Spark Structured Streaming for high-speed processing & Cassandra for storage.The final output is visualized at Streamlit Dashboard.

Astronomy Broker based on Apache Spark

Spark Structured Streaming data pipeline that processes movie ratings data in real-time.

Python package for working with demand-side grid projects, datasets and queries

This is my intro Repository that gives details about me and the work i do

MCP Server for Apache Spark History Server. The bridge between Agentic AI and Apache Spark.

Creating a Real-Time Flight-info Data Pipeline with Kafka, Apache Spark, Elasticsearch and Kibana

Flexible and scalable framework for data input and output operations in Spark applications. It offers a set of powerful tools and abstractions to simplify and streamline data processing pipelines.

A scalable marketing analytics pipeline built with Apache Spark and Delta Lake, designed to process, transform, and export data for advanced business insights.

Production-grade Basel III RWA calculation pipeline processing 120M+ records/day with Spark, Airflow, and AWS

Dataproc templates and pipelines for solving in-cloud data tasks

Production-grade CDC system processing 50M+ records/day with intelligent hybrid sync strategies

Real-time AML transaction monitoring system processing 5M transactions/day with <3s latency. Built with Kafka, Spark Streaming, Delta Lake, AWS Glue, and Airflow. Features: rule-based detection, SCD Type 2 customer profiling, automated compliance reporting, and comprehensive data quality framework.