Skip to content

About

OpenECPDS boosts the efficiency and productivity of data services by using proven and innovative technologies. It offers a portable, adaptable application for diverse environments, with a user-friendly tool for managing data acquisition, dissemination with push/pull mechanisms, and a notification system, all using standard protocols.

Topics

Resources

Contributing

Stars

12 stars

Watchers

14 watching

Forks

Latest commit

 

History

1,874 Commits

Folders and files

Repository files navigation

OpenECPDS

Our mission with OpenECPDS is to keep data moving.
Inspired by operational excellence. Powered by open-source innovation.
Acquire from anywhere. Deliver everywhere. Connect with confidence. Share without limits.


OpenECPDS is a mature, production-grade data management platform developed at ECMWF and distributed as open-source software to encourage collaboration.

Built for demanding meteorological operations, designed for any data domain. OpenECPDS is domain-agnostic, not a meteorology-specific platform: its core transfers, stores and distributes files without requiring a particular scientific format or subject matter. Research, industry, public services and enterprise teams can use it to automate provider ingestion, distribute datasets to partners, and offer self-service data access across on-premises and cloud environments.

See Why OpenECPDS? for its benefits, potential use cases and comparison with Managed File Transfer platforms.

It delivers three strategic services from a single system:

Service Description
Data Acquisition Automatic discovery and retrieval of data from remote providers
Data Dissemination Automatic distribution of data products to remote sites
Data Portal Pull/push access initiated by remote sites, with real-time monitoring

Built on container technologies, it scales from a laptop to hundreds of systems managing petabytes of data. It supports a wide range of protocols (FTP, SFTP, HTTP/S, WebDAV, S3, GCS, Azure Blob, MQTT/WIS2, …) and integrates with object storage, MQTT brokers, and Kubernetes.

OpenECPDS in ten minutes — from first launch to real workflows

A narrated introduction to the standalone environment, transfer controls, custom MQTT notifications, S3 access and the Monitor UI.

OpenECPDS-Standalone-v8.mp4

🚀 Try It Now — No Build Required

Experience the full OpenECPDS stack in a single command. The standalone image bundles MariaDB, Master, Data Mover, and Monitor — everything pre-configured and ready to go.

Requirements: 4 GB RAM and 6 GB free disk recommended. See System Requirements for details.

docker run -d \
  --name standalone \
  -v $(pwd)/ecpds-data:/data \
  -p 7443:7443 \
  -p 7022:7022 \
  -p 8443:8443 \
  -p 8883:8883 \
  -p 127.0.0.1:8080:8080 \
  -p 127.0.0.1:9404:9404 \
  -p 127.0.0.1:9405:9405 \
  -p 127.0.0.1:9406:9406 \
  ghcr.io/ecmwf/open-ecpds/standalone:latest

The database initialises automatically on first start. The /data volume persists everything across restarts.

Note: The container uses a self-signed TLS certificate — accept the browser security warning on first visit.

Access

Service URL Credentials
Monitoring UI https://localhost:8443 admin / admin2021 · monitor / monitor2021 · operator / operator2021
Data Portal (HTTPS) https://localhost:7443 test / test2021
Data Portal (S3) https://localhost:7443/s3 test / test2021
Data Portal (WebDAV) https://localhost:7443/webdav test / test2021
Data Portal (SFTP) sftp://localhost:7022 test / test2021
MQTTS broker mqtts://localhost:8883 test / test2021
Hawtio (JMX console) https://localhost:8080 hawtio / hawtio2021 (Jolokia login: jolokia / jolokia2021)

Exposed ports

Port Service
7443 Data Mover — HTTPS portal
7022 Data Mover — SFTP
8883 Data Mover — MQTTS (MQTT over TLS)
8443 Monitor — HTTPS UI
9640 Master — ECpds CLI
8080 Hawtio — authenticated HTTPS JMX console (preset Master, Monitor and Mover connections)
9404 / 9405 / 9406 Prometheus JMX exporter — Master / Mover / Monitor

Note on FTP: OpenECPDS fully supports FTP in production deployments. However, FTP passive mode (PASV) is not compatible with Docker port mapping — the server advertises its internal container address for data connections, which external clients cannot reach. For that reason, FTP is disabled in this standalone image. Use SFTP (port 7022) as a drop-in alternative for file transfers in Docker.

Note on JMX monitoring: The standalone image includes Hawtio and the Prometheus JMX exporter (details). Point Prometheus/Grafana at http://localhost:9404/metrics (Master), :9405 (Mover) and :9406 (Monitor). The credentials above are for evaluation only, and the exporter endpoints are unauthenticated — keep these ports bound to localhost (as in the command above) and never expose them on an untrusted network.

Populate some data

The standalone image ships with a few example destinations (e.g. hourly_aq), but they start out empty and switched off. Before testing the protocols below, log into the Monitoring UI and start one or two of them so there is some data to experiment with. This requires the container to have outbound Internet access, as these destinations acquire data from external providers:

  1. Open https://localhost:8443 and log in as admin / admin2021.
  2. On the main page, in the Transmission section, select Destinations.
  3. Click a destination's name (e.g. hourly_aq) in the table to open its Data page.
  4. Click the green Start Destination (▶) button above the transfers table.

Once started, the destination retrieves files from its source host and makes them available through the Data Portal, so the commands below have something to list and download.

Testing the protocols

All examples below use the pre-configured test / test2021 account. The container uses a self-signed certificate — pass the appropriate insecure/skip-verify flag for each tool.

S3 — using AWS CLI:

aws configure set aws_access_key_id test
aws configure set aws_secret_access_key test2021
aws s3 ls s3:// --endpoint-url https://localhost:7443/s3 --no-verify-ssl

Or with rclone:

rclone lsd :s3: \
  --s3-provider=Other \
  --s3-endpoint=https://localhost:7443/s3 \
  --s3-access-key-id=test \
  --s3-secret-access-key=test2021 \
  --s3-v2-auth \
  --no-check-certificate

Note: --s3-v2-auth selects AWS Signature V2 authentication. Use this setting with the rclone example to avoid SignatureDoesNotMatch errors from incompatible V4 request signing.

WebDAV — using curl:

# List root directory (PROPFIND)
curl -k -u test:test2021 -X PROPFIND https://localhost:7443/webdav/

Or with rclone:

rclone lsd :webdav: \
  --webdav-url=https://localhost:7443/webdav \
  --webdav-vendor=other \
  --webdav-user=test \
  --webdav-pass=$(rclone obscure test2021) \
  --no-check-certificate

Or mount directly from your OS file manager:

  • macOS Finder: Go → Connect to Server… (⌘K), enter https://localhost:7443/webdav, click Connect, then log in with test / test2021.
  • Windows Explorer: Right-click This PC → Map network drive, choose a drive letter, enter https://localhost:7443/webdav as the folder, check Connect using different credentials and log in with test / test2021.
  • Linux (GNOME Files / Nautilus): Go → Connect to Server…, enter davs://localhost:7443/webdav, log in with test / test2021.
  • Linux (command-line): mount with davfs2: sudo mount -t davfs https://localhost:7443/webdav /mnt/webdav

Note: All OS file managers will warn about the self-signed certificate — accept it to proceed. The WebDAV share appears as a network drive alongside your local disks.

SFTP — using the sftp command-line client:

sftp -P 7022 -o StrictHostKeyChecking=no test@localhost
# Password: test2021

MQTTS — The broker uses TLS with a self-signed certificate.

Using mqttx CLI (recommended — supports true --insecure):

mqttx sub \
  -h localhost -p 8883 \
  -l mqtts \
  -u test -P test2021 \
  --insecure \
  -t '#'

Try MQTT upload notifications

The pre-configured mqtt_demo destination lets you try upload notifications without an external data provider. Leave this destination stopped in the Monitor: its standby settings keep uploaded files in HOLD, available for download without outbound delivery, and trigger MQTT publication.

In one terminal, subscribe with mqttx CLI:

mqttx sub \
  -h localhost -p 8883 \
  -l mqtts \
  -u test -P test2021 \
  --insecure \
  -t 'openecpds/demo/#'

Wait for a successful connection/subscription and leave it running. Messages are non-retained, so subscribing after publication will not replay them. --insecure is only for the evaluation container's self-signed certificate.

In a second terminal, create a small file and connect over SFTP:

printf 'Hello from the standalone MQTT demo!\n' > mqtt-demo.txt
sftp -P 7022 test@localhost

Enter password test2021, accept the host key after checking it is your evaluation container, then run at the SFTP prompt:

put mqtt-demo.txt /mqtt_demo/mqtt-demo.txt
bye

Allow roughly a minute for asynchronous publication. The subscriber receives JSON on openecpds/demo/mqtt-demo.txt, including authenticated download links and, for files up to 16 KiB, inline Base64 content. Base64 is not encryption. Verify the file through HTTPS:

curl --fail -k -u test:test2021 \
  https://localhost:7443/ecpds/data/file/mqtt_demo/mqtt-demo.txt

In the Monitor, open Operations → MQTT Activity and filter on mqtt_demo. ACCEPTED means broker acceptance, not proof of subscriber receipt. Use a different filename for another test upload.

See the full MQTT upload walkthrough for troubleshooting and public endpoint configuration. For a remote container, replace localhost in the commands and update the destination's public endpoint settings so notification links use the reachable hostname and mapped ports.

Submitting files with the ecpds CLI

The ecpds command-line client submits files directly to the Master Server (port 9640). It is included in the standalone image — no separate installation needed:

docker exec standalone ecpds \
  -host localhost -port 9640 \
  -user test -pass test2021 \
  -dest TEST \
  /path/to/myfile.dat

Alternatively, pull the dedicated CLI image and point it at the container:

# Pull the CLI image
docker pull ghcr.io/ecmwf/open-ecpds/cli:latest

# Submit a file (Linux — use --network host to reach the standalone container)
docker run --rm --network host \
  -v /path/to/myfile.dat:/myfile.dat \
  ghcr.io/ecmwf/open-ecpds/cli:latest \
  -host localhost -port 9640 \
  -user test -pass test2021 \
  -dest TEST \
  /myfile.dat

Or download a pre-compiled binary from the GitHub Releases page (ecpds-amd64 / ecpds-arm64) and run it directly — the binary only requires OpenSSL:

curl -Lo /usr/local/bin/ecpds \
  https://github.com/ecmwf/open-ecpds/releases/latest/download/ecpds-amd64
chmod +x /usr/local/bin/ecpds
ecpds -host localhost -port 9640 -user test -pass test2021 -dest TEST /path/to/myfile.dat

See Getting the ecpds CLI for the full list of options and installation methods.

Logs

# All services (supervisord output)
docker logs -f standalone

# Individual service logs
docker exec standalone tail -f /data/log/master/master.log
docker exec standalone tail -f /data/log/mover/mover.log
docker exec standalone tail -f /data/log/monitor/monitor.log

Note: The standalone image runs with DEBUG logging enabled by default. This produces verbose output that is useful for exploring how OpenECPDS processes transfers, connects to data movers, and handles incoming requests. In a production deployment the log level can be controlled via the LOG_LEVEL environment variable (e.g. LOG_LEVEL=warn).


📖 Documentation

Full documentation is published at ecmwf.github.io/open-ecpds.

Section Description
Home Platform overview, core capabilities and quick links
Why OpenECPDS Benefits, domain-agnostic use cases and comparison with Managed File Transfer platforms
Getting Started Requirements, standalone container, installation, first run, IDE setup, ECPDS CLI
Architecture Overview, centralised vs. federated, data ownership & catalogue, components, failover, data transfer lifecycle, continental data movers, data mover allocation & load balancing, database, Java dependencies
Deployment Kubernetes, physical infrastructure, container registry, releasing
Concepts Entities, host types, protocols, plugin infrastructure, object storage, destination/host/web-user options, additional features
Monitor UI Monitoring, destinations, hosts, transfer history, data files, transfer methods & modules, users, administration
Administration Service configuration, TLS certificates, mover control-channel security, TOTP, OpenID Connect, ECaccess identity, critical action password, purge, storage lifecycle policies, product descriptions & status messages, system messages, login page content, destination metadata
Service Configuration Master, Monitor and Data Mover ecmwf.properties reference (under Administration)
Use Cases CLI tool, data portal, data users, web users, data user options, acquisition, acquisition options, dissemination
Transfer Modules ECtrans common options, FTP, FTPS, SFTP, HTTP/S, WebDAV, S3, GCS, Azure Blob, ECauth, optional ECaccess gateways, Portal, Test
Host Directory Acquisition, dissemination, replication, source, backup, proxy scripts
Notifications (MQTT) Real-time dissemination notifications, automated MQTT acquisition, optional WMO WIS2 integration, implementation details
Monitoring Transfer network statistics, TCP socket metrics, JMX with Hawtio & Prometheus, delay notifications
Event Logging PRS, RET, UPH, INH, ERR, CPY, DEA event categories and field reference
REST API REST API reference (v1)
API Reference JavaDocs
Global Reach 1,000+ destinations across 80+ countries
Support Support materials and resources
Glossary Key terms and concepts
Changelog Release notes
Contributing How to contribute

🛠 Going Further — Build from Source

To build the platform from source and develop against it, you need Docker (with the default socket enabled). Tested on Linux, macOS (Intel/Apple Silicon), and Windows WSL 2.

Run make help at any time to list all available targets.

1 — Outside the development container

# Build the development container image, start it, and open a shell inside it
make dev

If the container already exists, you can reuse it without rebuilding, replace it after a successful build, or cancel. Replacement discards container-only files, not bind-mounted host files.

Alternatively, download a published dev image instead of building it:

make dev-pull

Run this on the host from the repository root. It detects the architecture (override with ARCH=amd64 or ARCH=arm64) and offers to reuse or replace an existing container, or cancel. This requires a published architecture tag. See the installation guide for authentication and runtime credential details.

2 — Inside the development container

# Compile Java sources, build RPM packages, and create Docker images
make build

# Start all OpenECPDS services
cd run/bin/ecpds && make up

The monitoring UI is then available at https://localhost:3443 and the data portal at https://localhost:4443.

Local images use the open-ecpds/* namespace. Containers use short names such as master, mover, monitor and standalone. Use service keys in Makefile commands, for example make logs svc=master. Registry images are published under ghcr.io/ecmwf/open-ecpds/*.

For optional local OpenTelemetry testing, run make start-otel from the repository root, then make logs-otel to inspect decoded logs, traces and metrics. Use make stop-otel when finished. This diagnostic Collector is not started by the commands above and is not durable telemetry storage. See local telemetry setup for JVM configuration and container networking.

See Getting Started for the full walkthrough, including log inspection, stopping services, IDE setup, and Kubernetes deployment.

Building the standalone image locally

# Inside the development container:
make build-sa

This stages the RPMs (including the ecpds CLI binary) into docker/ecpds/standalone/ and builds the open-ecpds/standalone:<tag> image locally.


📜 License

Copyright 2022–2026 ECMWF. Licensed under the Apache License 2.0.

About

OpenECPDS boosts the efficiency and productivity of data services by using proven and innovative technologies. It offers a portable, adaptable application for diverse environments, with a user-friendly tool for managing data acquisition, dissemination with push/pull mechanisms, and a notification system, all using standard protocols.

Topics

Resources

Contributing

Stars

12 stars

Watchers

14 watching

Forks

Releases

Packages

Used by

Contributors

Languages