A multi-layer Java library for validating and analysing Darwin Core Data Packages (DwC-DP), built on Frictionless Data Package v1 and DuckDB.

Validation and analysis runs in multiple layers:

Validates datapackage.json against the Frictionless Data Package v1 spec:

• File exists and is valid JSON
• resources array is non-empty
• Each resource has a name and its declared path exists on disk
• Foreign key reference.resource values resolve to declared resource names
• Field type values are from the Frictionless v1 vocabulary

Validates the descriptor against the bundled dwc-dp-profile.json schema via networknt json-schema-validator:

• Top-level profile is present and a valid URI
• Each DwC-DP named resource declares profile: tabular-data-resource
• Each field declares dcterms:isVersionOf as a valid URI

For each resource whose name matches a reserved DwC-DP table name, validates against the bundled canonical table schemas:

• All constraints.required=true fields are declared
• Declared field types match the canonical type
• Required foreign keys are declared

Validates the optional eml.xml file if present alongside datapackage.json:

• Well-formed XML
• Required <title> and <creator> elements present
• Conformance with the bundled EML 2.2.0 XSD schema

Runs directly over data files (CSV, TSV, Parquet) without loading data into JVM memory:

• Foreign key validation — NOT EXISTS checks for each schema.foreignKeys rule
• Primary key uniqueness — duplicate detection across declared primary key fields
• Data type validation — TRY_CAST checks for each typed field (integer, number, boolean, date, datetime, time, year, object, array)
• Column statistics — populated value counts and distinct value counts per field

All violation results include sample rows/values and structured JSON detail for machine consumption.

Every ValidationIssue carries:

Severity defaults are defined in DefaultSeverities and can be overridden per deployment by passing a Map<DescriptorViolationType, ValidationIssue.Severity> to any validator constructor.

mvn test

# Text output (default)
mvn -q exec:java \
  -Dexec.mainClass=org.gbif.dp.cli.ValidationCli \
  -Dexec.args="/absolute/path/to/datapackage.json"

# JSON output
mvn -q exec:java \
  -Dexec.mainClass=org.gbif.dp.cli.ValidationCli \
  -Dexec.args="/absolute/path/to/datapackage.json --output-format JSON"

The CLI exits with:

DuckDB resource usage can be tuned via CLI flags or environment variables:

• Data analysis runs entirely inside DuckDB with file-backed scans — no data is loaded into JVM memory.
• Only small violation samples are materialised in Java (default: 20 rows, configurable via ValidationOptions).
• Parquet resources are supported alongside CSV and TSV.
• CSV dialect (delimiter, quote character, escape character, null sequence) is read from the dialect descriptor field, with automatic fallback to tab-separated for .tsv extensions.