GeoParquet: worker reads every row and column, no pruning or projection #1028

Summary

The GeoParquet worker reads every row and every column of the file, then materialises the whole dataset twice. Row-group pruning and column projection are the main reasons to ship GeoParquet rather than GeoJSON, and we currently use neither.

Filing this to capture the options and, importantly, the constraint that rules the most obvious one out.

Current behaviour

src/gm3/components/map/layers/geoparquet/worker.js:

const metadata = await parquetMetadataAsync(file);
// ...
const data = await parquetReadObjects({ file, compressors });

const features = [];
for (const row of data) {
  // ...
  features.push({ type: "Feature", geometry, properties: scrubProperties(properties) });
}

Three costs:

1. The metadata is parsed twice. We call parquetMetadataAsync(file) ourselves, then parquetReadObjects does options.metadata ??= await parquetMetadataAsync(options.file, options) because we never pass ours in.
2. Every column is read, whether or not anything references it.
3. Both full arrays are alive at once — data (all rows as objects) and features (the same rows again as GeoJSON) — before either can be released.

What hyparquet already supports

parquetRead / parquetReadObjects accept metadata, columns, rowStart, rowEnd, filter and onChunk, and the docs note it "reads the minimal number of row groups and columns to satisfy the request".

The metadata also carries per-row-group geospatial statistics (hyparquet/src/metadata.js):

geospatial_statistics: {
  bbox: { xmin, xmax, ymin, ymax, zmin, zmax, mmin, mmax },
  geospatial_types: ...,
}

That is a bounding box per row group, which is what spatial pruning needs.

Column projection cannot be inferred

This is the part worth recording, because it looks like the easy win and is not.

There is no reliable static list of which properties a layer uses:

• auto templates have no field list. src/gm3/application.js renders them by iterating Object.keys(feature.properties). The properties are the template. Projecting columns would silently shrink an auto identify result with no error.
• remote templates are fetched lazily at query time. src/gm3/application.js does fetch(layerTemplate.src) when a query first needs the template, which is long after the data has loaded. Their contents cannot be scanned at load time. The desktop example's GeoParquet layer relies on this for both identify (./templates/parcels.html) and select-grid-columns (./templates/parcel-columns.json).
• alias templates indirect to another template, so resolution is not local.
• Templates can call out to application JS — the example's search template calls app.highlightFeatures({'PIN': ...}) — so arbitrary properties can be referenced at runtime.

getFilterFieldNames in src/gm3/util.js covers query filters, but that is only one of the consumers.

So if we want column projection it has to be explicit and opt-in, declared on the map-source in the mapbook, with the author accepting responsibility for keeping it in step with their templates. Anything inferred will silently drop fields.

Suggested work

Safe and self-contained:

1. Pass the metadata we already parsed into parquetReadObjects. One line, removes a redundant parse.
2. Stream instead of double-buffering. Build features incrementally via onChunk rather than holding the full row array and the full feature array at the same time.

Needs a design decision:

3. Opt-in column projection, e.g. a <columns> element on the map-source. Only worth it if someone has a real wide-table case; the failure mode when it drifts out of sync with a template is silent.
4. Row-group pruning by bbox using geospatial_statistics. Note this conflicts with the current architecture: the layer uses strategy: all and the feature store holds one complete copy that queries run against, so pruning rows spatially would make search and select miss features outside the loaded area. It only pays off if loading becomes viewport- or query-driven, which is a larger change and probably its own issue.

Items 1 and 2 can be done now. 3 and 4 should not be started without agreeing the above first.

Related: #1026 (declared CRS is ignored).