SchemaElement¶

dev.hardwood.metadata.SchemaElement represents one element in the flat schema list in a Parquet footer. The list uses depth-first order.

FileSchema.fromSchemaElements(List<SchemaElement>) builds a schema tree from this list. FileSchema.toSchemaElements() creates the list again.

Note

Use FileSchema.builder(String) to build a schema for writing. The builder checks the complete schema and calculates child counts. Use SchemaElement when you work with footer metadata.

Factories¶

The static factories build the common element kinds.

Factory	Builds
`root(name, numChildren)`	the root group, with no repetition
`group(name, repetition, numChildren)`	a group
`group(name, repetition, numChildren, logicalType)`	a group with a logical type
`primitive(name, type, repetition)`	a primitive column
`primitive(name, type, repetition, logicalType)`	a primitive column with a logical type
`fixedLengthPrimitive(name, typeLength, repetition)`	a `FIXED_LEN_BYTE_ARRAY` column
`fixedLengthPrimitive(name, typeLength, repetition, logicalType)`	a `FIXED_LEN_BYTE_ARRAY` column with a logical type

import static dev.hardwood.metadata.SchemaElement.fixedLengthPrimitive;
import static dev.hardwood.metadata.SchemaElement.primitive;
import static dev.hardwood.metadata.SchemaElement.root;

List<SchemaElement> elements = List.of(
        root("schema", 3),
        primitive("id", PhysicalType.INT64, RepetitionType.REQUIRED),
        primitive("name", PhysicalType.BYTE_ARRAY, RepetitionType.OPTIONAL, LogicalType.string()),
        fixedLengthPrimitive("uuid", 16, RepetitionType.REQUIRED, LogicalType.uuid()));

FileSchema schema = FileSchema.fromSchemaElements(elements);

Repetition¶

The Parquet format leaves repetition_type unset on the root element and requires it on every other element.

root builds the root element with no repetition; the other factories require one.

fromSchemaElements accepts an element with no repetition value, as a footer may carry one. It uses REQUIRED for the root and OPTIONAL for other elements. FileSchema.toSchemaElements() writes REQUIRED for the root of a schema built with FileSchema.builder(String).

Name¶

The Thrift name field is required, but a malformed or truncated footer can omit it; the reader then creates an element with name == null. The factories pass a null name through without a check.

Type length¶

typeLength has a different meaning for each physical type:

Physical type	Meaning
`FIXED_LEN_BYTE_ARRAY`	the byte length of each value
any other physical type	the maximum bit length needed to store any value

fixedLengthPrimitive sets the physical type to FIXED_LEN_BYTE_ARRAY and takes the byte length as its second argument.

The maximum bit length has no factory. Use the canonical constructor for it:

// An INT32 column with a maximum bit length of 3 - all values of this column can be stored with 3 bits.
SchemaElement tag = new SchemaElement("tag", PhysicalType.INT32, 3, RepetitionType.REQUIRED,
        null, null, null, null, null, null);

typeLength can be null in raw footer metadata. A writer must provide a positive length for a FIXED_LEN_BYTE_ARRAY column. A data reader also needs this length to decode its values. ParquetFileWriter.create refuses a schema from fromSchemaElements that has a FIXED_LEN_BYTE_ARRAY column without a positive length, a REQUIRED column annotated UNKNOWN, or another annotation the writer cannot write, as FileSchema.Builder refuses them.

Errors¶

The factories throw IllegalArgumentException for these inputs:

Condition	Factory
`repetition` is `null`	`group`, `primitive`, `fixedLengthPrimitive`
`type` is `null`	`primitive`
`type` is `FIXED_LEN_BYTE_ARRAY`	`primitive`
`numChildren` is negative	`group`, `root`
`typeLength` is zero or less	`fixedLengthPrimitive`

A factory checks one element; fromSchemaElements checks the complete list.

Canonical constructor¶

The record constructor takes the components in Hardwood record-component order:

new SchemaElement(name, type, typeLength, repetitionType, 
                  numChildren, convertedType, scale, precision, 
                  fieldId, logicalType);

Use the constructor for metadata the factories do not set:

a legacy ConvertedType annotation, such as ConvertedType.LIST or ConvertedType.MAP;
legacy decimal metadata with scale and precision;
a Thrift fieldId;
a typeLength on a column that is not FIXED_LEN_BYTE_ARRAY;
an element with no repetition value that is not the root;
a complete element decoded from a footer.

Node kind¶

Method	Returns `true` when
`isGroup()`	`type` is `null`
`isPrimitive()`	`type` is not `null`