Skip to content

Hardwood

A modern lightweight Java reader and writer for the Apache Parquet file format. Available as a Java library and a command-line tool.

What's new

Why Hardwood

Hardwood gives applications fast and efficient support for reading and writing Parquet, without pulling in Hadoop, Avro, or the wider parquet-java dependency tree. It is built to be:

  • Light-weight: Zero transitive dependencies beyond optional compression libraries (Snappy, ZSTD, LZ4, Brotli)
  • Fast: Hardwood aims to be the fastest Parquet reader and writer for the JVM (1.0 read benchmarks)
  • Complete: Read and write support for flat and nested schemas, every logical type, every primitive type in current use, and the encodings and codecs in current use, with new format additions tracked as they land
  • Scalable: Hardwood is multi-threaded at the core, pages are decoded in parallel, with cross-file prefetching for multi-file reads
  • Embeddable: The Hardwood library can be used in GraalVM native binaries; a WASM build of dive runs in the browser as a preview
  • Agent-friendly: Hardwood's CLI comes with a skill which lets your agents inspect and analyse Parquet files
  • Compatible: A drop-in shim module facilitates migration from parquet-java, with documented divergences where Hardwood applies stricter semantics (e.g. SQL three-valued notEq)

The CLI inspects and analyses Parquet files and includes an interactive TUI for exploring a file's schema, row groups, pages, and data.

Quick Example

Reading a file with the row-based API:

import dev.hardwood.InputFile;
import dev.hardwood.reader.ParquetFileReader;
import dev.hardwood.reader.RowReader;

try (ParquetFileReader fileReader = ParquetFileReader.open(InputFile.of(path));
    RowReader rowReader = fileReader.rowReader()) {

    while (rowReader.hasNext()) {
        rowReader.next();

        long id = rowReader.getLong("id");
        String name = rowReader.getString("name");
        LocalDate birthDate = rowReader.getDate("birth_date");
        Instant createdAt = rowReader.getTimestamp("created_at");
    }
}

Writing a file row by row:

import dev.hardwood.OutputFile;
import dev.hardwood.writer.ParquetFileWriter;
import dev.hardwood.writer.RowWriter;

try (ParquetFileWriter writer = ParquetFileWriter.create(OutputFile.of(path), schema)) {
    RowWriter rows = writer.rowWriter();

    for (Person person : people) {
        rows.writeRow(row -> row
                .setLong("id", person.id())
                .setString("name", person.name())
                .setDate("birth_date", person.birthDate()));
    }
}

Set up Hardwood in your project, then read your first file end-to-end. Alternatively, install the Hardwood CLI for working with Parquet files on the command line.

Status and Limitations

The read path is stable and suitable for production use; ColumnReader, with its Validity and LayerKind types, is annotated @Experimental, so its shape may still change. The write path ships both APIs described under Choosing a Writer; the row-oriented RowWriter and its builders, the ColumnBatch setters for nested and nullable columns, and the in-memory destination InMemoryOutputFile are annotated @Experimental, so their shape may still change.

Writing targets local files through OutputFile.of(Path) and the heap through OutputFile.inMemory(). Reads have no file-size ceiling, but a column chunk read without an OffsetIndex is capped at 2 GB, and some backends carry their own limits; see Limits.

Roadmap

Forward-looking items, none committed to a specific release.

  • Finalize ColumnReader API — stabilize the API for columnar access and move it out of "Experimental" state. (#522)
  • Writer extensions — object-store output and parallel column encoding, on top of the write path described in The Write Model. (#9)
  • Parquet Modular Encryption — read files encrypted under the Parquet Modular Encryption spec: encrypted footer, per-column keys, AES-GCM and AES-GCM-CTR. (#128)
  • Apache Arrow interop — ColumnReader output as Arrow FieldVector / VectorSchemaRoot for zero-copy handoff to DuckDB, DataFusion, Pandas-via-JNI, and other Arrow-native consumers. (#153)

Getting help

Ask questions on GitHub Discussions and file bugs on the issue tracker; see Contributing.

Articles, talks & podcasts