Writer Reference¶
Destination¶
The OutputFile a writer is created with decides where the file is written.
| Factory | Destination |
|---|---|
OutputFile.of(Path) |
A local file. The bytes are streamed to a temporary sibling of the target path and renamed onto it when the writer closes, so a write that fails or is abandoned leaves nothing at the path |
OutputFile.inMemory() |
The heap, up to Integer.MAX_VALUE - 8 bytes. Returns an InMemoryOutputFile, whose buffer() returns the finished file |
An in-memory file can be written and read back without a filesystem: buffer() returns the whole file as a ByteBuffer positioned at its first byte, which is what InputFile.of(ByteBuffer) expects.
InMemoryOutputFile out = OutputFile.inMemory();
try (ParquetFileWriter writer = ParquetFileWriter.create(out, schema)) {
RowWriter rows = writer.rowWriter();
rows.writeRow(row -> row.setLong("id", 1L));
}
try (ParquetFileReader reader = ParquetFileReader.open(InputFile.of(out.buffer()))) {
// ...
}
The buffer grows with the file, so no size has to be given up front; the length of a Parquet file is only known once it has been written. The whole file is held on the heap, which costs the size of the finished file in addition to the memory the writer uses for the open row group. Each call to buffer() returns a buffer with its own position and limit, so reading the file back leaves it intact. The buffer's contents must not be modified; whether a change shows in the buffers other calls return is unspecified.
The file is complete only after the writer is closed, and buffer() throws IllegalStateException before that, or if the destination was discarded.
To write anywhere else (an object store, a network connection), implement OutputFile.
Schema¶
A file's schema is declared with FileSchema.builder(String) and built once; the same FileSchema is passed to ParquetFileWriter.create. Every method appends one field in declaration order, which is also the order the leaf-column indices follow.
| Method | Declares |
|---|---|
addColumn(name, type, repetition) |
a primitive column |
addColumn(name, type, repetition, logicalType) |
a primitive column carrying an annotation; a FIXED_LEN_BYTE_ARRAY column takes the length the annotation implies |
addColumn(name, type, repetition, typeLength) |
a FIXED_LEN_BYTE_ARRAY column of the given byte length |
addColumn(name, type, repetition, typeLength, logicalType) |
both of the above |
struct(name, repetition, filler) |
a struct group whose fields filler declares |
list(name, repetition, element) |
a LIST group whose element element declares |
map(name, repetition, keyType[, keyTypeLength][, keyLogicalType], value) |
a MAP group with a REQUIRED key of keyType and a value value declares |
build() |
the FileSchema |
The writer requires at least one column: build() rejects a schema with no fields, and ParquetFileWriter.create rejects one built another way, such as the childless root a file that declares no columns is read as.
repetition is REQUIRED or OPTIONAL; REPEATED is rejected, repetition being what list and map express. typeLength is rejected for every type but FIXED_LEN_BYTE_ARRAY, and a FIXED_LEN_BYTE_ARRAY column has a positive one. Declared without one, it takes the length its annotation implies:
| Annotation | Implied typeLength |
|---|---|
UUID |
16 |
INTERVAL |
12 |
FLOAT16 |
2 |
DECIMAL(precision, scale) |
the fewest bytes whose two's complement holds precision digits: 1 for precision 1–2, 4 for 9, 8 for 18, 16 for 38 |
Any other annotation, or none, implies no length, and the column is declared through a typeLength overload. A typeLength given explicitly is kept as given; a DECIMAL may take more bytes than its precision needs, while a UUID, INTERVAL or FLOAT16 of any other length is rejected.
FileSchema schema = FileSchema.builder("event")
.addColumn("id", PhysicalType.FIXED_LEN_BYTE_ARRAY, RepetitionType.REQUIRED,
LogicalType.uuid()) // 16 bytes
.addColumn("amount", PhysicalType.FIXED_LEN_BYTE_ARRAY, RepetitionType.OPTIONAL,
LogicalType.decimal(18, 2)) // 8 bytes
.addColumn("legacy_amount", PhysicalType.FIXED_LEN_BYTE_ARRAY, RepetitionType.OPTIONAL, 16,
LogicalType.decimal(18, 2))
.build();
Inside a struct filler the same methods appear on FileSchema.StructBuilder; a list element and a map value are declared on FileSchema.ElementBuilder, whose primitive(type, repetition[, typeLength][, logicalType]), struct, list and map carry no name, the surrounding layout supplying it.
Logical Type Factories¶
An annotation is built with the LogicalType static factory of the same name. The parameterless factories return a shared instance.
| Factory | Annotation |
|---|---|
LogicalType.string() |
STRING — UTF-8 encoded string |
LogicalType.enumType() |
ENUM — enum stored as a UTF-8 string |
LogicalType.uuid() |
UUID — 16-byte fixed-length byte array |
LogicalType.date() |
DATE — days since the Unix epoch |
LogicalType.json() |
JSON — JSON document stored as a UTF-8 string |
LogicalType.bson() |
BSON — BSON document stored as a byte array |
LogicalType.interval() |
INTERVAL — 12-byte fixed-length byte array (months, days, millis) |
LogicalType.float16() |
FLOAT16 — IEEE 754 half-precision, 2-byte fixed-length byte array |
LogicalType.nullType() |
NULL — every value in the column is null |
LogicalType.list() |
LIST |
LogicalType.map() |
MAP |
LogicalType.intType(bitWidth, isSigned) |
INT_8 … UINT_64 — bitWidth is 8, 16, 32 or 64 |
LogicalType.decimal(precision, scale) |
DECIMAL(precision, scale) — precision is positive and scale lies in [0, precision]; other values throw IllegalArgumentException |
LogicalType.time(isAdjustedToUTC, unit) |
TIME — unit is TimeUnit.MILLIS, MICROS or NANOS |
LogicalType.timestamp(isAdjustedToUTC, unit) |
TIMESTAMP — same units, over an INT64 or a FIXED_LEN_BYTE_ARRAY(12) |
LogicalType.variant(specVersion) |
VARIANT — specVersion is 1; the writer rejects a column carrying it |
LogicalType.geometry(crs) |
GEOMETRY — crs defaults to OGC:CRS84 when null |
LogicalType.geography(crs, edgeInterpolation) |
GEOGRAPHY — edgeInterpolation is an EdgeInterpolationAlgorithm |
Which Java value each annotation accepts is listed under Logical Types and Row Setters; a pairing of annotation and physical type the format does not define is rejected when the schema is built.
Writer Options¶
Write-time behaviour is configured with an immutable WriterConfig, passed to ParquetFileWriter.create(out, schema, config). WriterConfig.defaults() returns the defaults, which the two-argument create(out, schema) uses.
WriterConfig config = WriterConfig.builder()
.codec(CompressionCodec.ZSTD)
.rowGroupBufferTargetBytes(64L << 20)
.encoding("temperature", ColumnEncoding.BYTE_STREAM_SPLIT)
.build();
| Option | Default | Description |
|---|---|---|
pageTargetBytes(int) |
1 MiB |
Encoded bytes before a data page is cut, counting the values as the chunk's chosen encoding writes them, so a dictionary column is measured in indices. A ceiling: the page is cut before the record that would cross it, and only a single record larger than the whole target can breach it, a page holding whole records. Must be at least 4 bytes. |
pageTargetRows(int) |
20,000 |
Records before a data page is cut. A page holds whole records, so on a repeated column it counts records, not values. Sets how finely the page index lets a reader skip within a column chunk; it binds on columns whose values encode small, such as dictionary-encoded ones. Must be positive. |
rowGroupTargetRows(long) |
1,048,576 |
Records per row group: row groups hold exactly this many records apart from the last. Must be positive; a target above the structural ceiling of Integer.MAX_VALUE - 9 records is that ceiling, so Long.MAX_VALUE means "cut on bytes alone". |
rowGroupBufferTargetBytes(long) |
128 MiB |
Bytes the writer holds for the open row group before cutting it: level streams, dictionary indices, value stores and dictionaries. A row group passes it by at most one record. A row group is also cut before a repeated column's chunk would reach Integer.MAX_VALUE - 8 values and nulls, or before any column chunk would pass Integer.MAX_VALUE - 8 bytes of BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY values, so a target above 2 GiB can yield smaller row groups. See The Write Model. Must be positive. |
codec(CompressionCodec) |
ZSTD, or UNCOMPRESSED when the ZSTD library is absent |
Codec each page body is compressed with. |
encoding(ColumnEncoding) |
AUTO |
Encoding policy for every column without an override of its own. |
encoding(String, ColumnEncoding) |
— | Encoding policy for one leaf column, overriding the file-wide default. |
bloomFilter(String) |
— | Writes a Bloom filter for one leaf column at a false-positive probability of 0.01. A path naming no leaf column, or a BOOLEAN column, fails when the writer is created. See Bloom filters. |
bloomFilter(String, double) |
— | Writes a Bloom filter for one leaf column at the given false-positive probability. Must be strictly between 0 and 1. |
statisticsTruncationLength(int) |
64 |
Longest BYTE_ARRAY min / max statistics bound. A longer bound is truncated and flagged inexact. Must be positive. |
precisionLossPolicy(PrecisionLossPolicy) |
REJECT |
What the row-oriented layer does with a value carrying more precision than its column can hold. |
A page is cut at whichever of the two page targets is reached first, and a row group at whichever of the two row-group targets is reached first.
Each option has a getter that reads the configured value back: pageTargetBytes(), pageTargetRows(), rowGroupTargetRows(), rowGroupBufferTargetBytes(), codec(), statisticsTruncationLength(), precisionLossPolicy(), for the two encoding setters, defaultEncoding() and columnEncodings(), and for the Bloom filter setters, bloomFilters(), a map from column path to probability. Every setter rejects null, and the numeric bounds above are checked when the option is set.
Encodings¶
ColumnEncoding sets how a column's values are encoded, file-wide or per leaf column. AUTO leaves the choice to the writer; every other member names an encoding outright, and a column that names one builds no dictionary.
| Policy | What it writes |
|---|---|
AUTO |
Per column chunk, whichever of a dictionary or PLAIN is smaller, measured once the row group is buffered. |
PLAIN |
Values as they are: fixed-width types little-endian, BYTE_ARRAY behind a 4-byte length prefix. |
DELTA_BINARY_PACKED |
Differences between neighbouring values, bit-packed per block. Suits sorted or slow-moving integers. |
DELTA_LENGTH_BYTE_ARRAY |
Lengths delta-encoded ahead of the values, cheaper than PLAIN's per-value length prefix where the lengths are similar. |
DELTA_BYTE_ARRAY |
Each value carrying only what it does not share with the value before it. Suits sorted values with common prefixes. |
BYTE_STREAM_SPLIT |
A value's bytes scattered across one stream per byte position. Changes no page's size on its own; it changes how well the codec afterwards compresses floating-point data. |
An encoding is legal only for some physical types:
| Policy | Physical types |
|---|---|
AUTO, PLAIN |
every writable type |
DELTA_BINARY_PACKED |
INT32, INT64 |
DELTA_LENGTH_BYTE_ARRAY |
BYTE_ARRAY |
DELTA_BYTE_ARRAY |
BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY |
BYTE_STREAM_SPLIT |
INT32, INT64, FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY |
A policy naming a column whose type cannot carry it fails when the writer is created, including a file-wide BYTE_STREAM_SPLIT over a schema holding a BYTE_ARRAY column. A per-column override names the column by its dotted leaf path as the schema spells it, synthetic segments included: readings.list.element, not readings. A path matching no leaf column is rejected the same way.
No policy demands a dictionary; it is an outcome AUTO may arrive at. A BOOLEAN column is never dictionary-encoded, so AUTO resolves it to PLAIN whatever its values are.
Compression Codecs¶
| Codec | Produced | Library needed |
|---|---|---|
UNCOMPRESSED |
yes | — |
GZIP |
yes | — (JDK) |
SNAPPY |
yes | org.xerial.snappy:snappy-java |
ZSTD |
yes | com.github.luben:zstd-jni |
LZ4_RAW |
yes | at.yawk.lz4:lz4-java |
BROTLI |
yes | com.aayushatharva.brotli4j:brotli4j |
LZ4 |
no | — |
LZO |
no | — |
A missing library, and either of the two refused codecs, fails when the writer is created rather than mid-file. LZ4 and LZO are refused on write. LZ4 files are read; LZO files are not.
Physical Types and Columnar Setters¶
Every physical type Parquet defines is written except INT96, which is deprecated and rejected when the writer is created. Each ColumnBatch setter comes in four shapes: by index or name, all-present or nullable (a Validity or a boolean[] mask).
| Physical type | ColumnBatch setter |
Values |
|---|---|---|
BOOLEAN |
booleans |
boolean[] |
INT32 |
ints |
int[] |
INT64 |
longs |
long[] |
FLOAT |
floats |
float[] |
DOUBLE |
doubles |
double[] |
BYTE_ARRAY |
bytes |
byte[][] |
FIXED_LEN_BYTE_ARRAY |
fixed |
byte[][], each exactly the declared length |
The columnar API takes physical values and converts nothing: a STRING column is written through bytes(...) as UTF-8, a DATE column through ints(...) as days since the Unix epoch. Nesting is described through struct(path, Validity), list(path, offsets[, Validity]) and map(path, offsets[, Validity]).
Logical Types and Row Setters¶
The row-oriented layer takes physical values through the setter named for the type, and converts logical-type values to the column's physical representation, taking the same Java types the reader returns. The physical setters apply whatever the column's annotation is: an INT32 column annotated DATE accepts setDate and setInt alike, the one taking a LocalDate and the other the epoch day it converts to.
| Logical type | Physical type | StructBuilder setter |
Java type |
|---|---|---|---|
| none | BOOLEAN |
setBoolean |
boolean |
| any | INT32 |
setInt |
int |
| any | INT64 |
setLong |
long |
| none | FLOAT |
setFloat |
float |
| none | DOUBLE |
setDouble |
double |
STRING, ENUM, JSON, or none |
BYTE_ARRAY |
setString |
String (written as UTF-8) |
| any | BYTE_ARRAYFIXED_LEN_BYTE_ARRAY |
setBinary |
byte[] |
DATE |
INT32 |
setDate |
LocalDate |
TIME |
INT32INT64 |
setTime |
LocalTime |
TIMESTAMP(isAdjustedToUTC = true) |
INT64FIXED_LEN_BYTE_ARRAY(12) |
setTimestamp |
Instant |
TIMESTAMP(isAdjustedToUTC = false) |
INT64FIXED_LEN_BYTE_ARRAY(12) |
setLocalTimestamp |
LocalDateTime |
DECIMAL |
INT32INT64BYTE_ARRAYFIXED_LEN_BYTE_ARRAY |
setDecimal |
BigDecimal |
UUID |
FIXED_LEN_BYTE_ARRAY(16) |
setUuid |
UUID |
INTERVAL |
FIXED_LEN_BYTE_ARRAY(12) |
setInterval |
PqInterval |
| any | any primitive | setNull |
— |
none / LIST / MAP |
group | setStruct / setList / setMap |
a filler over StructBuilder / ListBuilder / MapBuilder |
ListBuilder has the same set as addInt, addString, addStruct, addNull, … and MapBuilder has addEntry, whose entry struct declares the two fields key and value.
Timestamp columns split by isAdjustedToUTC, mirroring the reader's split between getTimestamp and getLocalTimestamp: using the wrong setter throws. Both APIs also accept every setter by field index; see Addressing fields by index.
A timestamp column's physical type sets the range of values it holds; its unit sets the precision on either:
| Unit | INT64 holds |
FIXED_LEN_BYTE_ARRAY(12) holds |
|---|---|---|
MILLIS |
about 292 million years either side of 1970 | every Instant and LocalDateTime |
MICROS |
about 292,000 years either side of 1970 | every Instant and LocalDateTime |
NANOS |
1677-09-21T00:12:43.145224192 to 2262-04-11T23:47:16.854775807 | every Instant and LocalDateTime |
A value outside the range of an INT64 column is rejected under either PrecisionLossPolicy.
Value Ranges¶
Some annotations narrow what their physical type may hold. Where one does, both write APIs reject a stored value outside that range, naming the offending row or field:
| Annotation | Accepted values |
|---|---|
INT(n) signed, n < physical width |
[-2^(n-1), 2^(n-1)) |
INT(n) unsigned, n < physical width |
[0, 2^n) |
TIME(unit, _) |
[0, one day in unit) — the annotation defines the value as elapsed time after midnight |
DECIMAL(p, s) |
an unscaled value of at most p digits |
UNKNOWN |
no value at all; every row must be null, so the column is set through a setter taking a null mask |
Every other annotation, DATE, TIMESTAMP, INT(32) and INT(64) included, admits every value of its physical type, and its column is not scanned per value. A large unsigned value is spelled as a negative, which is also how the reader returns it.
Two checks are not annotations and always apply: a FIXED_LEN_BYTE_ARRAY value must be exactly the length the column declares, and a present value of a binary column must not be null. The value at a row a Validity marks null is never encoded, so it is never checked.
An annotation over binary content is not a range, and the writer does not inspect it. A value passed to bytes(...) or fixed(...) is written as given, whether the column is annotated STRING, ENUM, JSON, BSON, VARIANT, GEOMETRY or GEOGRAPHY. Encoding a STRING column's values as UTF-8, and producing well-formed payloads under the others, is the caller's to do. Bytes that are not valid UTF-8 are written and read back as replacement characters rather than rejected. StructBuilder.setString takes a String and encodes it, so the row-oriented layer cannot produce that.
PrecisionLossPolicy governs precision only, and only on the row-oriented layer:
| Policy | Behaviour |
|---|---|
REJECT (default) |
Reject a value carrying digits the column's unit or scale cannot hold, naming the field. A value that is exact at the column's unit or scale is written normally. |
TRUNCATE |
Drop the digits that do not fit: a TIME / TIMESTAMP value is floored to the column's unit, a DECIMAL rescaled with RoundingMode.DOWN. |
A value the column cannot represent at all, such as a LocalDate beyond the INT32 day range or an unscaled decimal wider than the declared precision, is rejected under either policy.
Statistics Written¶
Every column chunk carries null_count. min / max are computed under the column's ColumnOrder during encoding and written to the preferred min_value / max_value fields; the deprecated min / max fields are not written.
- Bounds are omitted where their order is undefined. A column annotated
INTERVAL,UNKNOWN,VARIANT,GEOMETRY,GEOGRAPHY,LISTorMAPwrites its null count alone. An all-null chunk has no bounds to write either. - Truncation.
BYTE_ARRAYbounds longer thanstatisticsTruncationLengthare truncated and flagged inexact (is_min_value_exact/is_max_value_exact=false). A column annotatedSTRING,ENUMorJSONis truncated at a character boundary, so its bounds are valid UTF-8. Fixed-width types write those flags astrue. nan_countis written for everyFLOAT,DOUBLEandFLOAT16chunk, including when it is zero. No other type writes the field.distinct_countis written where the chunk still knows its cardinality exactly: any chunk underAUTOthat interned its values to the end, whichever encoding the flush-time comparison then chose, and anyBOOLEANchunk, which knows its cardinality without a dictionary. It is absent for any other chunk written under a named encoding, and for one that stopped interning part-way (see How the Writer Picks an Encoding).
Every column chunk also carries encoding_stats: its page count per page type and encoding. A dictionary-encoded chunk lists one PLAIN dictionary page and all of its data pages as RLE_DICTIONARY, so a reader can prune its row group on the dictionary for an eq or in predicate (see Query Controls). Any other chunk lists its data pages under the one value encoding it was written in.
Every file carries a page index, written between the last row group and the footer:
OffsetIndex, for every column chunk: the location of each data page and the first record it holds. Every page starts at a record boundary.ColumnIndex, for every column chunk whose bounds can be stated: each page'smin/maxunder the same rules as the chunk statistics, its null count, whether it holds only nulls, itsNaNcount forFLOAT,DOUBLEandFLOAT16, and whether the pages' bounds ascend or descend. A column without an order writes none, nor does a floating-point column chunk with a page whose values are allNaN. Where no truncatedmaxexists, a page'smaxis written whole.
The level histograms of SizeStatistics and the ColumnIndex, and the GeospatialStatistics of a GEOMETRY or GEOGRAPHY column are not written. Bounding-box pushdown prunes row groups from that last field, so it prunes nothing in a file Hardwood produced.
Bloom filters¶
To let eq and in predicates on a high-cardinality column skip row groups, add a Bloom filter for the column with bloomFilter(...) (see Writer Options). The column then carries a split-block Bloom filter in every row group. Bloom filters: skipping on equality describes which columns benefit and what a filter costs.
- Contents. Every non-null value of the column chunk, hashed with XXH64 over its bytes as stored:
INT32andFLOATas 4 little-endian bytes,INT64andDOUBLEas 8,BYTE_ARRAYwithout its length prefix,FIXED_LEN_BYTE_ARRAYas is.FLOATandDOUBLEvalues are hashed over their raw bits, so-0.0and0.0are distinct values, as areNaNs with different bits. - Size. About 1.2 bytes per distinct value of the chunk at the default probability of
0.01, rounded up to a power of two between 32 bytes and 128 MiB. Every chunk gets a filter whatever its encoding, an all-null chunk included. - Placement. All filters of the file are written together after the last row group, before the page index, with
bloom_filter_offsetandbloom_filter_lengthset on each column chunk.
File Metadata¶
Two footer fields are set on the ParquetFileWriter itself, at any point until close() writes the footer: the file's key-value metadata and its created_by identifier.
| Method | Description |
|---|---|
keyValueMetadata(String key, String value) |
Adds one key-value entry, replacing any value already held for that key. |
keyValueMetadata(Map<String, String> metadata) |
Adds every entry of the map, leaving entries it does not name in place. |
createdBy(String createdBy) |
Replaces the created_by identifier. |
try (ParquetFileWriter writer = ParquetFileWriter.create(out, schema)) {
writer.keyValueMetadata("ARROW:schema", encodedArrowSchema);
writer.columnWriter().writeBatch(batch -> batch.ints(0, values));
writer.keyValueMetadata("row.count", String.valueOf(values.length));
}
Key-value metadata¶
key_value_metadata is application-defined: Parquet does not interpret it, and the writer validates nothing beyond requiring a key. It is where ARROW:schema, the pandas descriptor, org.apache.spark.sql.parquet.row.metadata and the table-format stamps live.
Entries reach the file in the order they were added. A null value writes a key carrying no value, which the format allows and which is what a reader reports as a null, so the map FileMetaData.keyValueMetadata() returns can be passed straight to keyValueMetadata(Map) to reproduce another file's metadata exactly. A file given no entries carries no key_value_metadata field at all.
Reading the field back is covered in Read File Metadata.
created_by¶
The default identifier follows the convention Parquet readers parse, naming the library, its version and the build it came from:
ParquetFileWriter.DEFAULT_CREATED_BY holds it. createdBy(String) replaces it; readers that key compatibility workarounds off this field expect the <app> version <version> (build <hash>) shape, and a bare application name is rejected by some of them.
Ending a Row Group¶
ParquetFileWriter.endRowGroup() writes the open row group, so the next record starts a new one. Records staged by rowWriter() are written into the row group being closed. When no record was written since the last row group closed, it writes nothing.
The row-group targets close row groups as well; a row group ends at a target or at a call, whichever comes first. A row group closed by a call is written exactly as one a target closes after the same records.
endRowGroup() and close() throw IllegalStateException when called from inside a writeBatch or writeRow filler. Any exception from endRowGroup() fails the writer.
Finishing and Abandoning a File¶
| Call | Effect |
|---|---|
close() |
Writes the buffered row group and the footer, and publishes the file at the destination |
close() after the writer has failed |
Discards the output, leaving nothing at the destination |
abort() |
Discards the output and closes the writer |
close() or abort() on a closed or aborted writer |
Nothing |
The writer fails when ColumnWriter.writeBatch, RowWriter.writeRow or endRowGroup throws, whatever the exception: a batch or record rejected by the checks under What the Writer Rejects, an exception thrown by the filler, a destination IOException or a codec failure. RowWriter.tryWriteRow fails the writer only on the exceptions listed under RowWriteResult. A failed writer rejects further writes; keyValueMetadata and createdBy stay callable until close().
A failure while close() finishes or publishes the file discards the output as well, and close() throws it. When the output cannot be discarded, abort() and close() on a failed writer throw the IOException; a failure while finishing or publishing carries it as a suppressed exception.
RowWriteResult¶
RowWriter.tryWriteRow returns one of two members:
| Member | Meaning |
|---|---|
Staged |
The record is in the batch. A later flush can still fail the writer. |
Rejected(fieldPath, message) |
The record was not staged, and the staged batch is unchanged. fieldPath is the schema path the message names; message is the full text writeRow throws for the same rejection. |
A rejection that depends on the value handed over becomes Rejected: a value outside the range its annotation declares, a value whose magnitude or precision the column cannot hold under the configured PrecisionLossPolicy, a FIXED_LEN_BYTE_ARRAY value of the wrong length, or a REQUIRED field left unset or set null.
Every other failure throws and fails the writer, as it does under writeRow: an unknown field name or index, a field set twice, a setter that does not fit the field (including any value setter on a column annotated UNKNOWN), an exception thrown by the filler, and a failure while a completed batch is written.
What the Writer Rejects¶
| Exception | When |
|---|---|
UnsupportedOperationException |
A schema column of an unsupported physical type (INT96); a refused codec (LZ4, LZO), one whose library is missing, or one whose native library will not load; a schema shape the writer cannot produce |
IllegalArgumentException |
A schema with no columns; a null metadata key, metadata map or created_by; an unknown column name or path; a setter that does not fit the column's type; a null value array, or a null value at a present row of a binary column; a column set twice in one batch or record; a batch that leaves a column unset, or whose arrays disagree in length; a null mask on a REQUIRED column; a boolean[] mask whose length does not match the values; list offsets that do not start at 0, are not non-decreasing, or disagree with the element count; a value outside the range its annotation declares; a REQUIRED field left unset by a record; a record whose values for one column pass what a column chunk can hold; a Bloom filter configured for a BOOLEAN column, or with a false-positive probability outside (0, 1) |
IndexOutOfBoundsException |
A leaf-column index outside [0, leaf column count) on a ColumnBatch setter, or a field index outside [0, getFieldCount()) on a StructBuilder setter |
IllegalStateException |
Writing, ending a row group, or setting key-value metadata or created_by, after close(); writing after the writer has failed; using both write APIs on one file; ending a row group or closing the writer from inside a batch or record filler; using a ColumnBatch after it has been submitted, or a nested builder after its filler has returned; taking InMemoryOutputFile.buffer() before the writer has closed, or from a destination that was discarded |
IOException |
The destination cannot be created, written, or finalized; an in-memory file would pass Integer.MAX_VALUE - 8 bytes |
ParquetWriteException |
The file could not be produced for a reason that is neither the caller's input nor the destination: a compression codec that rejects a page body. Unchecked |
Schema Shapes¶
Repetition is writable wherever a LIST or MAP annotation accounts for it. An annotated group is REQUIRED or OPTIONAL and holds exactly one REPEATED entry, which for a MAP is a group of key and value. That admits the canonical three-level LIST and two-level MAP layouts FileSchema.builder declares, and the legacy two-level lists a schema read from an existing file may carry: LIST { repeated element }, whose entry is the element, and LIST { repeated group element { … } }, whose entry is an element struct.
Every other arrangement of repetition is rejected when the writer is created, since nothing in the schema says where its entries begin and end:
- a
REPEATEDfield, leaf or group, whose parent carries neither annotation; - a
LISTorMAPgroup that is itselfREPEATED; - a
LISTorMAPgroup holding anything other than its singleREPEATEDentry; - a
MAPwhose entry is a leaf rather than a group.
The row API reaches a list's values through an element node below the entry, which the legacy two-level lists do not have, so rowWriter() refuses those two shapes and columnWriter() writes them. rowWriter() also requires sibling field names to be unique, which the ColumnBatch indices and dotted paths do not.
A column's maximum definition level and maximum repetition level must each be at most 255. A schema nested deeper is rejected with UnsupportedOperationException when the writer is created.