ETLPlus

ETLPlus is a Python package and command-line interface for practical data extraction, validation, transformation, and loading workflows.

• Getting Started
• At a Glance
• Release Status
• Features
• Installation
• Quickstart
  • Command-line interface
  • Python API
• Support ETLPlus
• Data Connectors
  • REST APIs (api)
  • Databases (database)
  • Files (file)
    • Handler Matrix Guardrail
    • Stubbed / Placeholder
    • Tabular & Delimited Text
    • Semi-Structured Text
    • Columnar / Analytics-Friendly
    • Binary Serialization and Interchange
    • Databases and Embedded Storage
    • Spreadsheets
    • Statistical / Scientific / Numeric Computing
    • Logs and Event Streams
    • Data Archives
    • Templates
• Usage
  • Command Line Interface
    • Command Shapes
    • Initialize A Starter Project
    • Check Pipelines
    • Render SQL DDL
    • Extract Data
    • Validate Data
    • Transform Data
    • Inspect Run History
    • Schedule Configured Runs
    • Load Data
  • Python API
  • Complete ETL Pipeline Example
  • Format Overrides
• Transformation Operations
  • Filter Operations
  • Aggregation Functions
• Validation Rules
• Development
  • API Client Docs
  • Runner Internals and Connectors
  • Running Tests
    • Test Scope and Intent
  • Code Coverage
  • Linting
  • Updating Demo Snippets
  • Releasing to the Python Package Index (PyPI)
• License
• Contributing
• Documentation
  • Python Packages/Subpackage
  • Maintainer Docs
  • Community Health
  • Other
• Acknowledgments

ETLPlus helps you extract, validate, transform, and load data from files, databases, and APIs, either as a Python library or from the command line.

To get started:

• See Installation for setup instructions.
• See the Getting Started environment reference for the current environment-variable and provider-credential guidance.
• Try the Quickstart for a minimal working example (CLI and Python).
• Explore Usage for more detailed options and workflows.
• See SUPPORT.md for the current support policy, supported Python versions, and response targets.

ETLPlus currently supports Python 3.13 and 3.14.

• Install with pip install etlplus for the supported CLI, etlplus.ops, the API client, and the built-in implemented file handlers.
• Use pipx install etlplus or uv tool install etlplus when you want ETLPlus as an isolated CLI tool instead of a package installed into the current environment.
• Use pip install -e ".[dev]" for contributor tooling and pip install "etlplus[file]" when you need the remaining scientific and specialty format dependencies.
• Use uv sync --locked --extra dev when you want the committed uv.lock development snapshot instead of an unconstrained local resolver run.
• Use pip install "etlplus[storage]" when you want remote storage backends for s3://, azure-blob://, abfs://, or hdfs:// URIs through etlplus.storage and etlplus.file.File.
• Treat local paths, localhost databases, and Docker Compose helpers as development conveniences; the same config surface is designed to work with interchangeable remote backing services.
• Use secret:NAME or secret:env:NAME for environment-backed config secrets; keep secret:file:path.to.key as a local-development compatibility path while encrypted files and cloud secret backends remain deferred.
• Expect the most stable execution surface from the documented CLI commands, etlplus.ops, implemented file handlers, and etlplus.api.
• Use etlplus check --readiness --config <pipeline.yml> to confirm config substitution, optional extras, and common provider-credential hints before running remote-storage or cloud-database jobs.
• See docs/source/getting-started/compatibility.md for the supported Python versions, platform coverage, and dependency groups.
• See the Getting Started environment reference for the current runtime, storage, BigQuery, and Snowflake environment variables that readiness checks understand.
• See docs/source/getting-started/quickstart.md if you want the shortest path from install to a working ETL flow.

Detailed file-handler coverage and migration notes are still available later in this README and in docs/source/guides/file-handler-matrix.md, but they are no longer required reading to get started.

ETLPlus treats the v1.x line as its stable public release line. The repository still retains some placeholders, stubs, and migration-reference modules for historical or implementation reasons, but they are not part of the supported public contract unless they are explicitly documented as such.

The stable surface for the current v1.x releases is:

• The documented CLI commands: check, extract, history, init, load, log, render, report, run, status, transform, ui, and validate
• The documented Python ETL primitives in etlplus.ops, including the advanced step modules under etlplus.ops.transformations
• The implemented file handlers listed as implemented in the handler matrix
• The documented API client and pagination helpers under etlplus.api
• The read-only local run-history dashboard exposed through etlplus ui, backed by the same persisted history store used by the history, log, status, and report commands

The following are not part of the stable execution surface unless explicitly promoted later:

• Database extract/load execution paths that are still described as placeholders
• Stubbed file handlers and placeholder formats
• Defunct or migration-reference modules retained for historical context

Maintainers handling packaging, CI, versioned docs, or release gating should consult RELEASE-CHECKLIST.md.

Maintainer-facing policy and automation references are also available in RELEASE-POLICY.md, CI-CD-WORKFLOWS.md, and CONTRIBUTING.md.

• Init starter ETLPlus projects:

  • Scaffold a runnable file-to-file starter pipeline with sample input data
  • Get suggested next commands for checking and running the generated job

• Check data pipeline definitions before running them:

  • Summarize jobs, sources, targets, and transforms
  • Validate dependency graphs and print DAG order with --graph
  • Run lightweight runtime and config readiness checks with --readiness
  • Enable stricter diagnostics with --strict to catch malformed entries the tolerant loader would otherwise skip
  • Confirm configuration changes by printing focused sections on demand

• Render SQL DDL from shared table specs:

  • Generate CREATE TABLE or view statements
  • Swap templates or direct output to files for database migrations

• Extract data from multiple sources:

  • Files (CSV, JSON, XML, YAML)
  • Databases (connection string support; extract is a placeholder today)
  • REST APIs (GET)

• Validate data with flexible rules:

  • Type checking
  • Required fields
  • Value ranges (min/max)
  • String length constraints
  • Pattern matching
  • Enum validation

• Transform data with powerful operations:

  • Filter records
  • Map/rename fields
  • Select specific fields
  • Sort data
  • Aggregate functions (avg, count, max, min, sum)

• Load data to multiple targets:

  • Files (CSV, JSON, XML, YAML)
  • Databases (connection string support; load is a placeholder today)
  • REST APIs (PATCH, POST, PUT)

• Inspect local run history and reports:

  • List normalized runs with filters and table output
  • Stream raw append events for backend-level troubleshooting
  • Inspect the latest run or aggregate success and duration metrics by job, status, or day
  • Launch the optional read-only local history UI with etlplus ui

• Schedule configured runs without a resident ETLPlus daemon:

  • Inspect portable cron/interval schedule definitions from config
  • Emit crontab or systemd helper snippets for external schedulers
  • Dispatch currently due schedules once with etlplus schedule --run-pending

pip install etlplus

For an isolated CLI install, use either pipx or uv:

pipx install etlplus
uv tool install etlplus

Choose pip install etlplus when you want the ETLPlus library and CLI in the active Python environment. Choose pipx install etlplus for the preferred isolated CLI install or uv tool install etlplus when you manage command-line tools through uv.

These installer paths are first-class for the base CLI artifact. They install the same PyPI package as pip install etlplus, including the stable-line default dependency set.

Conda-forge status: the tagged PyPI sdist support gate has passed cross-platform recipe validation on Linux, macOS, and Windows. Until the conda-forge feedstock is accepted and published, use the PyPI-based installers above for supported installs.

For development:

pip install -e ".[dev]"

The default install includes the non-native dependencies used by the built-in file handlers for common binary, columnar, spreadsheet, and embedded-database formats such as cbor2, duckdb, fastavro, msgpack, openpyxl, odfpy, pandas, pyarrow, pymongo, xlrd, and xlwt.

This is intentional for the stable line. ETLPlus treats the documented CLI, etlplus.ops, etlplus.api, and the implemented built-in file handlers as one supported default runtime surface, so the base install keeps the dependencies needed for that surface together instead of pushing core implemented handlers behind extras.

The installer review keeps that dependency posture unchanged: pip, pipx, and uv tool install all install the same broad base runtime. A leaner default install would be a separate breaking-design decision for a future major release, not a minor v1.x change.

For development with full optional file-format support:

pip install -e ".[dev,file]"

For runtime-only optional file-format support:

pip install "etlplus[file]"

For runtime cloud-storage support:

pip install "etlplus[storage]"

For optional BigQuery connector metadata and readiness support:

pip install "etlplus[database-bigquery]"

For optional Snowflake connector metadata and readiness support:

pip install "etlplus[database-snowflake]"

The file extra is now reserved for the remaining scientific and specialty format dependencies such as netCDF4, pyreadr, pyreadstat, and xarray.

That split is also intentional: the file extra is reserved for narrower optional workflows rather than for the built-in formats that ETLPlus expects most users of the default runtime to have available.

The database-bigquery extra installs the optional packages used by the additive BigQuery database connector shape (provider: bigquery plus project and dataset when you are not supplying a connection string).

The database-snowflake extra installs the optional packages used by the additive Snowflake database connector shape (provider: snowflake plus account, database, and schema when you are not supplying a connection string).

See the Getting Started environment reference for the current environment-variable reference, environment-first secret tokens, and provider credential hints that etlplus check --readiness understands for S3, Azure storage, BigQuery, and Snowflake.

Get up and running in under a minute.

# Inspect help and version
etlplus --help
etlplus --version

# One-liner: extract CSV, filter, select, and write JSON
etlplus extract examples/data/sample.csv \
  | etlplus transform --operations '{"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}' \
  - temp/sample_output.json

from etlplus.ops import extract, transform, validate, load

data = extract("file", "input.csv")
ops = {"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}
filtered = transform(data, ops)
rules = {"name": {"type": "string", "required": True}, "email": {"type": "string", "required": True}}
assert validate(filtered, rules)["valid"]
load(filtered, "file", "temp/sample_output.json", file_format="json")

If ETLPlus saves you engineering time, consider supporting the project through the repository sponsor button once the funding links are live on the default branch. Funding helps to pay for:

• Maintenance and bug fixes
• New file, API, and database connectors
• Documentation, examples, and release automation
• Compatibility work for new Python and dependency versions

The preferred sponsorship path is GitHub Sponsors, with Buy Me a Coffee as the lightweight fallback for one-time support.

Support is only one way to contribute. ETLPlus also benefits from codeless contributions such as documentation fixes, issue triage, reproducible bug reports, usage feedback, examples, testing results, answering questions in discussions, and release validation.

For community participation, use GitHub Discussions for questions, docs feedback, examples, and support conversations. Use GitHub Issues for confirmed bugs and concrete feature work. See docs/community-discussions.md for the recommended setup.

Data connectors abstract sources from which to extract data and targets to which to load data. They are differentiated by their types, each of which is represented in the subsections below.

ETLPlus can extract from REST APIs and load results via common HTTP methods. Supported operations include GET for extract and PATCH/POST/PUT for load.

Database connectors use connection strings for extraction and loading, and DDL can be rendered from table specs for migrations or schema checks. Database extract/load operations are currently placeholders; plan to integrate a database client in your runner.

Managed database endpoints are the intended production shape for database connector configs. Use environment-injected credentials or provider metadata for BigQuery and Snowflake; keep localhost DSNs, SQLite files, and Docker-backed Postgres/MSSQL instances as development fixtures.

File connectors use the same type: file shape for local paths and remote object-storage URIs. Install the storage extra when you need first-class s3://, azure-blob://, abfs://, or hdfs:// locations; treat local filesystem paths as quick-iteration inputs and outputs.

Connector diagnostics share the same remediation wording in readiness and strict config checks. For secrets in connector configs, prefer secret:NAME or secret:env:NAME so values come from the runtime environment; reserve secret:file:path.to.key for local development files selected by ETLPLUS_SECRETS_FILE.

Recognized file formats are listed in the tables below. Support for reading to or writing from a recognized file format is marked as:

• Y: implemented (may require optional dependencies)
• N: stubbed or not yet implemented

Handler Architecture

• File IO is moving to class-based handlers rooted at etlplus/file/base.py (FileHandlerABC, category ABCs, and ReadOnlyFileHandlerABC).
• etlplus/file/registry.py resolves handlers using an explicit FileFormat -> handler class map.
• Dispatch is explicit-only: unmapped formats raise Unsupported format.
• Module-level etlplus.file.<format>.read() / write() wrapper APIs have been removed.
• Use handler instances directly (for example, JsonFile().read(path) / JsonFile().write(path, data)) or etlplus.file.File dispatch via File(path, file_format).read() and .write(...).
• Documentation and examples intentionally use handler class methods, not deprecated module wrappers.
• Placeholder handlers are split into:
  • etlplus/file/stub.py for generic stub behavior
  • etlplus/file/_stub_categories.py for category-aware internal stub ABCs
• Scientific/statistical handlers dta, nc, rda, rds, sav, and xpt now implement ScientificDatasetFileHandlerABC dataset hooks.

Current Migration Coverage (Class-Based + Explicit Registry Mapping)

• Delimited/text: csv, dat, fwf, psv, tab, tsv, txt
• Semi-structured/config: ini, json, ndjson, properties, toml, xml, yaml
• Columnar: arrow, feather, orc, parquet
• Binary/interchange: avro, bson, cbor, msgpack, pb, proto
• Embedded DB: duckdb, sqlite
• Spreadsheets: ods, xls, xlsm, xlsx
• Scientific/statistical: dta, nc, rda, rds, sav, xpt, sas7bdat (read-only), plus single-dataset scientific stubs mat, sylk, zsav
• Archive wrappers: gz, zip
• Log/event streams: log
• Templates: hbs, jinja2, mustache, vm
• Explicit module-owned stub handlers (via stub.py + _stub_categories.py): stub, accdb, cfg, conf, ion, mdb, numbers, pbf, wks

The concise matrix below is the migration guardrail for class-based handler coverage. For batch-by-batch maintenance notes and the same matrix in docs, see docs/file-handler-matrix.md.

ETLPlus provides a powerful CLI for ETL operations:

# Show help
etlplus --help

# Show version
etlplus --version

The CLI is implemented with Typer (Click-based). The legacy argparse parser has been removed, so rely on the documented commands/flags and run etlplus <command> --help for current options.

The core commands accept positional source and target arguments when you want to read from or write to explicit paths or URIs. When you omit them, ETLPlus falls back to standard streams:

• extract: etlplus extract [SOURCE]
  • Omit SOURCE to read from STDIN.
• transform: etlplus transform [SOURCE] [TARGET]
  • Omit SOURCE to read from STDIN and omit TARGET to write to STDOUT.
• load: etlplus load [TARGET]
  • Omit TARGET to write to STDOUT.
• validate: etlplus validate [SOURCE]
  • Omit SOURCE to read from STDIN and use --output if you want file output instead of STDOUT.

Use --source-format, --target-format, --source-type, and --target-type to override the usual inference rules when a filename, URI, or stream does not provide enough context.

Use etlplus init to scaffold a minimal starter project with a sample pipeline and input data:

etlplus init demo-pipeline
cd demo-pipeline
etlplus check --config pipeline.yml --jobs
etlplus run --config pipeline.yml --job file_to_file_customers

Use etlplus check to explore pipeline YAML definitions without running them. The command can print job names, summarize configured sources and targets, drill into specific sections, or run readiness checks.

Inspect config contents:

etlplus check --config examples/configs/pipeline.yml --jobs
etlplus check --config examples/configs/pipeline.yml --summary

Show sources or transforms for troubleshooting:

etlplus check --config examples/configs/pipeline.yml --sources
etlplus check --config examples/configs/pipeline.yml --transforms

Run runtime and config readiness checks:

etlplus check --readiness
etlplus check --readiness --config examples/configs/pipeline.yml
etlplus check --readiness --strict --config examples/configs/pipeline.yml

Readiness warnings are advisory and still return exit code 0. Fatal readiness errors, such as unresolved required environment variables, missing blocking optional dependencies, or provider bootstrap failures, return exit code 1.

Validate dependency order before executing a DAG-shaped pipeline:

etlplus check --config examples/configs/pipeline.yml --graph

Use etlplus render to turn table schema specs into ready-to-run SQL. Render from a pipeline config or from a standalone schema file, and choose the built-in ddl or view templates (or provide your own).

Render all tables defined in a pipeline:

etlplus render --config examples/configs/pipeline.yml --template ddl

Render a single table in that pipeline:

etlplus render --config examples/configs/pipeline.yml --table customers --template view

Render from a standalone table spec to a file:

etlplus render --spec schemas/customer.yml --template view -o temp/customer_view.sql

Note: For file sources, the format is normally inferred from the filename extension. Use --source-format to override inference when a file lacks an extension or when you want to force a specific parser.

Extract from JSON file:

etlplus extract examples/data/sample.json

Extract from CSV file:

etlplus extract examples/data/sample.csv

Extract from XML file:

etlplus extract examples/data/sample.xml

Extract from REST API:

etlplus extract https://api.example.com/data

Save extracted data to file:

etlplus extract examples/data/sample.csv > temp/sample_output.json

Validate data from file or JSON string:

etlplus validate '{"name": "John", "age": 30}' --rules '{"name": {"type": "string", "required": true}, "age": {"type": "number", "min": 0, "max": 150}}'

Validate from file:

etlplus validate examples/data/sample.json --rules '{"email": {"type": "string", "pattern": "^[\\w.-]+@[\\w.-]+\\.\\w+$"}}'

Validate JSON or YAML against a JSON Schema:

etlplus validate examples/data/sample.json --schema examples/schemas/customer.schema.json --schema-format jsonschema
etlplus validate --source-format yaml --schema examples/schemas/pipeline.schema.json --schema-format jsonschema -

When the source or schema path already makes the schema family clear, the CLI can infer it without --schema-format:

etlplus validate examples/data/sample.json --schema examples/schemas/customer.schema.json
etlplus validate examples/data/sample.xml --schema examples/data/sample.xsd

Validate CSV against a Frictionless Table Schema:

etlplus validate data/customers.csv --schema examples/schemas/customers.table-schema.json --schema-format frictionless

Inference rules are intentionally narrow and predictable:

• An explicit --schema-format always wins
• .xsd schemas resolve to XSD validation
• JSON or YAML source hints resolve to JSON Schema validation
• CSV source hints resolve to Frictionless validation
• Ambiguous inline or STDIN cases require --schema-format

CSV schema failures preserve row and field paths in the same result envelope:

{
  "valid": false,
  "errors": [
    "row[3].email: Row at position \"3\" has unique constraint violation in field \"email\" at position \"1\": the same as in the row at position 2",
    "row[3].status: The cell \"\" in row at position \"3\" and field \"status\" at position \"2\" does not conform to a constraint: constraint \"required\" is \"True\""
  ],
  "field_errors": {
    "row[3].email": [
      "Row at position \"3\" has unique constraint violation in field \"email\" at position \"1\": the same as in the row at position 2"
    ],
    "row[3].status": [
      "The cell \"\" in row at position \"3\" and field \"status\" at position \"2\" does not conform to a constraint: constraint \"required\" is \"True\""
    ]
  },
  "data": null
}

When piping data through etlplus transform, use --source-format whenever the SOURCE argument is - or a literal payload, mirroring the etlplus extract semantics. When TARGET is omitted or set to -, etlplus transform emits JSON to STDOUT. When TARGET is a file path or file URI, the transformed payload is written directly. When TARGET is an API or database target and you provide --target-type, the command delegates the transformed payload to etlplus load and prints the downstream load result envelope. --target-format affects file targets and delegated load targets that honor a format hint. Use --source-type to override the inferred source connector type and --target-type to override the inferred target connector type, matching the etlplus extract/ etlplus load behavior.

Transform file inputs while overriding connector types:

etlplus transform \
  --operations '{"select": ["name", "email"]}' \
  examples/data/sample.json  --source-type file \
  temp/selected_output.json --target-type file

Filter and select fields:

etlplus transform \
  --operations '{"filter": {"field": "age", "op": "gt", "value": 26}, "select": ["name"]}' \
  '[{"name": "John", "age": 30}, {"name": "Jane", "age": 25}]'

Sort data:

etlplus transform \
  --operations '{"sort": {"field": "age", "reverse": true}}' \
  examples/data/sample.json

Aggregate data:

etlplus transform \
  --operations '{"aggregate": {"field": "age", "func": "sum"}}' \
  examples/data/sample.json

Map/rename fields:

etlplus transform \
  --operations '{"map": {"name": "new_name"}}' \
  examples/data/sample.json

Send transformed data to a REST API through the load path:

etlplus transform \
  --operations '{"select": ["name", "email"]}' \
  examples/data/sample.json \
  https://api.example.com/customers --target-type api

Database targets use the same delegated load path, but the current database load implementation is still a documented placeholder.

etlplus run persists local run history keyed by run_id. DAG-aware runs keep a compact aggregate summary on the parent run row and also persist one per-job history row for each executed job. Use the read/query commands to inspect that history without opening the backend directly.

Scheduler-triggered runs dispatched through etlplus schedule --run-pending reuse the same history path and add scheduler metadata additively under result_summary.scheduler.

List recent normalized runs:

etlplus history --job file_to_file_customers --status succeeded --limit 10 --table

List recent normalized job rows from DAG-aware runs:

etlplus history --level job --pipeline customer_sync --limit 10 --table

Show the latest matching run:

etlplus status --job file_to_file_customers

Show the latest matching job row:

etlplus status --level job --job file_to_file_customers

Stream raw run-level history events:

etlplus log --run-id 8e4a33d7 --follow

Stream raw job-level history events:

etlplus log --level job --pipeline customer_sync --status skipped --follow

Aggregate grouped history metrics:

etlplus report --group-by day --since 2026-03-01T00:00:00Z --table

Aggregate per-job history by pipeline:

etlplus report --level job --group-by pipeline --since 2026-03-01T00:00:00Z --table

Use etlplus schedule to inspect schedule config, emit OS helper snippets, or dispatch due schedules one time while reusing the normal etlplus run execution path.

Inspect configured schedules:

etlplus schedule --config examples/configs/scheduling.yml

Emit a systemd helper for one schedule:

etlplus schedule --config examples/configs/scheduling.yml --schedule hourly_sync --emit systemd

Dispatch currently due schedules and forward structured lifecycle events:

etlplus schedule --config examples/configs/scheduling.yml --run-pending --event-format jsonl

--run-pending is intentionally one-shot. Invoke it from cron, systemd, or CI if you want recurring execution without adding a resident ETLPlus scheduler process.

etlplus load consumes JSON from STDIN; provide only the target argument plus optional flags.

Load to JSON file:

etlplus extract examples/data/sample.json \
  | etlplus load temp/sample_output.json --target-type file

Load to CSV file:

etlplus extract examples/data/sample.csv \
  | etlplus load temp/sample_output.csv --target-type file

Load to REST API:

cat examples/data/sample.json \
  | etlplus load https://api.example.com/endpoint --target-type api

Use ETLPlus as a Python library:

from etlplus.ops import extract, validate, transform, load

# Extract data
data = extract("file", "data.json")

# Validate data
validation_rules = {
    "name": {"type": "string", "required": True},
    "age": {"type": "number", "min": 0, "max": 150}
}
result = validate(data, validation_rules)
if result["valid"]:
    print("Data is valid!")

# Transform data
operations = {
    "filter": {"field": "age", "op": "gt", "value": 18},
    "select": ["name", "email"]
}
transformed = transform(data, operations)

# Load data
load(transformed, "file", "temp/sample_output.json", file_format="json")

For YAML-driven pipelines executed end-to-end (extract → validate → transform → load), see:

• Authoring: docs/pipeline-guide.md
• Runner API and internals: see etlplus.ops.run docstrings and docs/pipeline-guide.md.

CLI quick reference for pipelines:

# List jobs or show a pipeline summary
etlplus check --config examples/configs/pipeline.yml --jobs
etlplus check --config examples/configs/pipeline.yml --summary
etlplus check --config examples/configs/pipeline.yml --graph

# Run a job
etlplus run --config examples/configs/pipeline.yml --job file_to_file_customers

# Run every configured job in DAG order
etlplus run --config examples/configs/pipeline.yml --all

# Run a job and emit structured events to STDERR
etlplus run --config examples/configs/pipeline.yml --job file_to_file_customers --event-format jsonl

Structured events use the stable etlplus.event.v1 envelope. Additive fields may appear over time, but breaking field/lifecycle changes require a schema version bump.

# 1. Extract from CSV
etlplus extract examples/data/sample.csv > temp/sample_extracted.json

# 2. Transform (filter and select fields)
etlplus transform \
  --operations '{"filter": {"field": "age", "op": "gt", "value": 25}, "select": ["name", "email"]}' \
  temp/sample_extracted.json \
  temp/sample_transformed.json

# 3. Validate transformed data
etlplus validate \
  --rules '{"name": {"type": "string", "required": true}, "email": {"type": "string", "required": true}}' \
  temp/sample_transformed.json

# 4. Load to CSV
cat temp/sample_transformed.json \
  | etlplus load temp/sample_output.csv

--source-format and --target-format override whichever format would normally be inferred from a file extension. This is useful when an input lacks an extension (for example, records.txt that actually contains CSV) or when you intentionally want to treat a file as another format.

Examples (zsh):

# Force CSV parsing for an extension-less file
etlplus extract data.txt --source-type file --source-format csv

# Write CSV to a file without the .csv suffix
etlplus load output.bin --target-type file --target-format csv < data.json

# Leave the flags off when extensions already match the desired format
etlplus extract data.csv --source-type file
etlplus load output.json --target-type file < data.json

Supported operators:

• eq: Equal
• ne: Not equal
• gt: Greater than
• gte: Greater than or equal
• lt: Less than
• lte: Less than or equal
• in: Value in list
• contains: List/string contains value

Example:

{
  "filter": {
    "field": "status",
    "op": "in",
    "value": ["active", "pending"]
  }
}

Supported functions:

• sum: Sum of values
• avg: Average of values
• min: Minimum value
• max: Maximum value
• count: Count of values

Example:

{
  "aggregate": {
    "field": "revenue",
    "func": "sum"
  }
}

Supported validation rules:

• type: Data type (string, number, integer, boolean, array, object)
• required: Field is required (true/false)
• min: Minimum value for numbers
• max: Maximum value for numbers
• minLength: Minimum length for strings
• maxLength: Maximum length for strings
• pattern: Regex pattern for strings
• enum: List of allowed values

Schema-based validation is also supported through etlplus validate --schema .... Use --schema-format xsd for XML documents, --schema-format jsonschema for JSON or YAML documents, and --schema-format frictionless for CSV documents. When the file path already makes the schema family unambiguous, ETLPlus can infer it; ambiguous inline text and STDIN cases still require an explicit schema format.

Example:

{
  "email": {
    "type": "string",
    "required": true,
    "pattern": "^[\\w.-]+@[\\w.-]+\\.\\w+$"
  },
  "age": {
    "type": "number",
    "min": 0,
    "max": 150
  },
  "status": {
    "type": "string",
    "enum": ["active", "inactive", "pending"]
  }
}

Looking for the HTTP client and pagination helpers? See the dedicated docs in etlplus/api/README.md for:

• Quickstart with EndpointClient
• Authentication via EndpointCredentialsBearer
• Pagination with PaginationConfig (page and cursor styles)
• Tips on records_path and cursor_path

Curious how the pipeline runner composes API requests, pagination, and load calls?

• Runner overview and helpers: see etlplus.ops.run docstrings and docs/pipeline-guide.md
• Unified "connector" vocabulary (API/File/DB): etlplus/connector
  • API/file targets reuse the same shapes as sources; API targets typically set a method.

For local CI parity and full coverage of remaining optional file formats, install:

pip install -e ".[dev,file]"

# Lightweight run (uses currently installed extras)
pytest

# Full run with remaining optional file-format dependencies
make test-full

ETLPlus organizes tests by scope and uses markers for cross-cutting intent.

• Scope folders:
  • Unit (tests/unit/): isolated function/class behavior, no external services.
  • Integration (tests/integration/): cross-module and boundary behavior.
  • E2E (tests/e2e/): full workflow/system-boundary behavior.
• Intent markers:
  • smoke: go/no-go viability checks.
  • contract: interface/metadata compatibility checks.

Smoke tests are now treated as an intent marker rather than a primary folder. The legacy path migration is complete; smoke tests live under scope folders and are selected by marker.

If a test calls etlplus.cli.main() or etlplus.ops.run.run(), it is integration by default. Detailed criteria and marker conventions: CONTRIBUTING.md#testing, tests/README.md.

pytest tests/unit tests/integration tests/e2e --cov=etlplus --cov-report=html

make lint
make doclint
make fmt
make typecheck

make lint runs the Ruff-based source checks used in CI, make doclint runs pydocstyle and pydoclint, make fmt applies the supported Ruff-plus-autopep8 formatting path, and make typecheck runs mypy against the shipped package. ETLPlus no longer maintains separate Black or Flake8 contributor paths; Ruff is the authoritative lint gate and autopep8 remains as the compatibility formatter used by CI and pre-commit. .ruff.toml is the canonical line-length source, and any duplicated formatter width in supporting tooling is expected to match it. If an external tool still invokes Flake8, the repository .flake8 file exists only as a compatibility shim for the overlapping basics that Flake8 can understand.

DEMO.md shows the real output of etlplus --version captured from a freshly built wheel. Regenerate the snippet (and the companion file docs/snippets/installation_version.md) after changing anything that affects the version string:

make demo-snippets

The helper script in tools/update_demo_snippets.py builds the wheel, installs it into a throwaway virtual environment, runs etlplus --version, and rewrites the snippet between the markers in DEMO.md.

setuptools-scm derives the package version from Git tags, so publishing is now entirely tag driven—no hand-editing pyproject.toml, setup.py, or etlplus/__version__.py.

GitHub Releases is the canonical release-history surface for ETLPlus. It is also the earlier developer-preview and release-announcement surface for tagged releases, whereas PyPI is the later public package-install channel. The docs changelog page links there, and the maintainer-facing release text is drafted from the template and category config in the .github/ folder.

1. Ensure main is green and the release notes/docs are up to date.

2. Create and push a SemVer tag matching the v*.*.* pattern:

   git tag -a v1.4.0 -m "Release v1.4.0"
   git push origin v1.4.0

3. GitHub Actions runs the tagged release workflow in .github/workflows/cd.yml, builds the sdist/wheel, validates the artifacts, validates the tagged docs build, publishes the GitHub Release, and then publishes to PyPI.

4. Draft the GitHub Release notes using .github/RELEASE-NOTES-TEMPLATE.md together with the categorized notes configured in .github/release.yml.

The tagged docs publication itself is handled by the Read the Docs GitHub App after the tag push; the release workflow only validates that the docs build cleanly from the tagged source.

If you want an extra smoke-test before tagging, run make dist && pip install dist/*.whl locally; this exercises the same build path the workflow uses.

This project is licensed under the MIT License.

Code and codeless contributions are welcome! If you’d like to add a new feature, fix a bug, or improve the documentation, please feel free to submit a pull request as follows:

1. Fork this repository.
2. Create a new feature branch for your changes (git checkout -b feature/feature-name).
3. Commit your changes (git commit -m "Add feature").
4. Push to your branch (git push origin feature-name).
5. Submit a pull request with a detailed description.

If you choose to be a code contributor, please first refer these documents:

• Pipeline authoring guide: docs/pipeline-guide.md
• Design notes (Mapping inputs, dict outputs): docs/pipeline-guide.md#design-notes-mapping-inputs-dict-outputs
• Typing philosophy (TypedDicts as editor hints, permissive runtime): CONTRIBUTING.md#typing-philosophy

Valuable non-code contributions include:

• Improving or correcting documentation
• Reporting bugs with clear reproduction steps
• Testing releases and platform-specific behavior
• Proposing examples, tutorials, and workflow patterns
• Answering questions in GitHub Discussions
• Sponsoring the project through GitHub Sponsors or Buy Me a Coffee

Navigate to detailed documentation for each subpackage:

• etlplus.api: Lightweight HTTP client and paginated REST helpers
• etlplus.cli: Command-line interface definitions for etlplus
• etlplus.database: Database engine, schema, and ORM helpers
• etlplus.file: Unified file format support and helpers
• etlplus.storage: Storage location parsing and backend helpers
• etlplus.ops: Extract/validate/transform/load primitives
• etlplus.templates: SQL and DDL template helpers
• etlplus.workflow: Helpers for data connectors, pipelines, jobs, and profiles

• Release Policy And Versioning: Release expectations, semantic-version-style rules, and deprecation posture
• CI/CD Workflow Map: Explanation of pr.yml, ci.yml, and cd.yml
• Release Checklist And Stable-Line Maintenance: Release-readiness execution guidance and stable-line follow-up tracking
• Maintainer Runbooks: Maintainer workflow guidance for protected branches and tag-driven releases

• Contributing Guidelines: How to contribute, report issues, and submit PRs
• Code of Conduct: Community standards and expectations
• Security Policy: Responsible disclosure and vulnerability reporting
• Support: Where to get help

• API client docs: etlplus/api/README.md
• Examples: examples/README.md
• File handler matrix guardrail: docs/file-handler-matrix.md
• Pipeline authoring guide: docs/pipeline-guide.md
• Runner internals: see etlplus.ops.run docstrings and docs/pipeline-guide.md
• Design notes (Mapping inputs, dict outputs): docs/pipeline-guide.md#design-notes-mapping-inputs-dict-outputs
• Typing philosophy: CONTRIBUTING.md#typing-philosophy
• Demo and walkthrough: DEMO.md
• Additional references: REFERENCES.md

ETLPlus is inspired by common work patterns in data engineering and software engineering patterns in Python development, aiming to increase productivity and reduce boilerplate code. Feedback and contributions are always appreciated!