Open-source data infrastructure
Krisztián Szűcs
For over a decade I've built the open-source infrastructure that data runs on — Apache Arrow, Parquet, Iceberg — with a focus on moving and storing data more efficiently.
Open Source Engineer at Hugging Face and an Apache Arrow PMC member. I'm one of the authors of Ibis, an Apache OpenDAL committer, and a regular contributor across the data ecosystem — with hundreds of merged pull requests to Arrow, DataFusion, Iceberg, Delta, and more.
What I work on
Columnar formats
Arrow, Parquet, Iceberg — the on-disk and in-memory layouts most of the modern data stack is built on. Understanding and improving how they work is most of what I do.
Moving and storing data efficiently
Content-defined chunking, content-addressable storage, dedup, object stores — moving bytes only when they actually change.
Rust and Mojo for data engines
Increasingly where the systems layer of the data stack lives. I build infrastructure in both.
Selected work
- Apache Arrow arrow · c++
Years of core work as a PMC member — the cross-platform CI/CD that ships packages at 100M+ downloads a month, PyArrow, and the Arrow C++ internals much of the columnar ecosystem is built on.
- Ibis python
One of the authors of Ibis, the portable Python dataframe library — its core, plus koerce, the pattern-matching and rewrite engine underneath it.
- marrow mojo
A ground-up Apache Arrow implementation in Mojo.
- Parquet Content-Defined Chunking parquet
A technique I designed and shipped: placing Parquet page boundaries by content instead of byte offset, so a barely-changed file re-uploads only the chunks that moved. It landed in PyArrow and pandas and powers Xet content-addressable storage — cutting data transfer by 90%+ in common rewrite patterns.
Across the ecosystem
Hundreds of merged pull requests over the past decade, across the projects the open data stack runs on: