DEV Community

#dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
What Actually Happens When You Call spark.read? One Line of Python, a Thousand Tasks
anikethsdeshpande profile

What Actually Happens When You Call spark.read? One Line of Python, a Thousand Tasks

#spark #dataengineering #python #distributedsystems
Comments
42 min read
An Analytics Agent's Permissions Should Survive a Bad Prompt
lingikaushikreddy profile

An Analytics Agent's Permissions Should Survive a Bad Prompt

#ai #security #database #dataengineering
Comments
2 min read
Building a Data Audit Workbench That Holds Up in an Assessment
maref profile

Building a Data Audit Workbench That Holds Up in an Assessment

#compliance #dataengineering #audit
Comments
3 min read
How to Build an Open Lakehouse on Your Laptop
lisancao profile
lnc

How to Build an Open Lakehouse on Your Laptop

#opensource #dataengineering #architecture #infrastructure
Comments
42 min read
A bad batch loaded with zero errors. One row added $4.5 million.
jigonyoo profile

A bad batch loaded with zero errors. One row added $4.5 million.

#dataengineering #dbt #dataquality #python
Comments
5 min read
Best Snowflake Data Modeling Tools: SQLDBM, erwin, Hackolade & Vertabelo for Reverse Engineering and DDL Generation
naveensub profile
Naveen

Best Snowflake Data Modeling Tools: SQLDBM, erwin, Hackolade & Vertabelo for Reverse Engineering and DDL Generation

#database #dataengineering #sql #tools
Comments
11 min read
Data Engineering Failures Cost Enterprises $3M a Month
datadriven profile

Data Engineering Failures Cost Enterprises $3M a Month

#dataengineering #career #devops #database
Comments
8 min read
Counting a phrase across the King James Bible — and why "fear not" is not 365 times
makeinterview profile

Counting a phrase across the King James Bible — and why "fear not" is not 365 times

#data #dataengineering
Comments
3 min read
The Distributed Compute Playground: Everything You Can Plug In and Tune in an Apache Spark Job
anikethsdeshpande profile

The Distributed Compute Playground: Everything You Can Plug In and Tune in an Apache Spark Job

#spark #dataengineering #python #distributedsystems
2
Comments 1
50 min read
Why LLMs Fall for Manipulation? - Victor Amit
victoramit profile

Why LLMs Fall for Manipulation? - Victor Amit

#ai #security #machinelearning #dataengineering
1
Comments 1
22 min read
Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew
firfircelik profile

Two PySpark contracts Staff DEs learn on a bad Tuesday: foreachBatch restarts and applyInPandas skew

#databricks #pyspark #python #dataengineering
1
Comments
8 min read
Hello Everyone
ayush_shandilya_dev profile

Hello Everyone

#ai #dataengineering #machinelearning #softwareengineering
Comments
1 min read
Building an Agentic Data Factory with Parquet, DuckDB, MCP, and Refinement Loops
kyleledbetter profile

Building an Agentic Data Factory with Parquet, DuckDB, MCP, and Refinement Loops

#ai #dataengineering #analytics #factory
Comments
5 min read
Stop bad data at the pull request with Great Expectations and GitHub Actions
laura_cristinachicovisd profile

Stop bad data at the pull request with Great Expectations and GitHub Actions

#dataengineering #githubactions #dataquality #python
Comments
4 min read
Reproducible Payroll Starts With a Versioned Calculation Manifest
sourcebento profile

Reproducible Payroll Starts With a Versioned Calculation Manifest

#architecture #python #dataengineering #ai
Comments
7 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.