Our mission with OpenECPDS is to keep data moving.
Inspired by operational excellence. Powered by open-source innovation.
Acquire from anywhere. Deliver everywhere. Connect with confidence. Share without limits.
OpenECPDS is a mature, production-grade data management platform developed at ECMWF and distributed as open-source software to encourage collaboration.
Built for demanding meteorological operations, designed for any data domain. OpenECPDS is domain-agnostic, not a meteorology-specific platform: its core transfers, stores and distributes files without requiring a particular scientific format or subject matter. Research, industry, public services and enterprise teams can use it to automate provider ingestion, distribute datasets to partners, and offer self-service data access across on-premises and cloud environments.
See Why OpenECPDS? for its benefits, potential use cases and comparison with Managed File Transfer platforms.
It delivers three strategic services from a single system:
| Service | Description |
|---|---|
| Data Acquisition | Automatic discovery and retrieval of data from remote providers |
| Data Dissemination | Automatic distribution of data products to remote sites |
| Data Portal | Pull/push access initiated by remote sites, with real-time monitoring |
Built on container technologies, it scales from a laptop to hundreds of systems managing petabytes of data. It supports a wide range of protocols (FTP, SFTP, HTTP/S, WebDAV, S3, GCS, Azure Blob, MQTT/WIS2, …) and integrates with object storage, MQTT brokers, and Kubernetes.
A narrated introduction to the standalone environment, transfer controls, custom MQTT notifications, S3 access and the Monitor UI.
OpenECPDS-Standalone-v8.mp4
Experience the full OpenECPDS stack in a single command. The standalone image bundles MariaDB, Master, Data Mover, and Monitor — everything pre-configured and ready to go.
Requirements: 4 GB RAM and 6 GB free disk recommended. See System Requirements for details.
docker run -d \
--name standalone \
-v $(pwd)/ecpds-data:/data \
-p 7443:7443 \
-p 7022:7022 \
-p 8443:8443 \
-p 8883:8883 \
-p 127.0.0.1:8080:8080 \
-p 127.0.0.1:9404:9404 \
-p 127.0.0.1:9405:9405 \
-p 127.0.0.1:9406:9406 \
ghcr.io/ecmwf/open-ecpds/standalone:latestThe database initialises automatically on first start. The /data volume persists everything across restarts.
Note: The container uses a self-signed TLS certificate — accept the browser security warning on first visit.
| Service | URL | Credentials |
|---|---|---|
| Monitoring UI | https://localhost:8443 |
admin / admin2021 · monitor / monitor2021 · operator / operator2021 |
| Data Portal (HTTPS) | https://localhost:7443 |
test / test2021 |
| Data Portal (S3) | https://localhost:7443/s3 |
test / test2021 |
| Data Portal (WebDAV) | https://localhost:7443/webdav |
test / test2021 |
| Data Portal (SFTP) | sftp://localhost:7022 |
test / test2021 |
| MQTTS broker | mqtts://localhost:8883 |
test / test2021 |
| Hawtio (JMX console) | https://localhost:8080 |
hawtio / hawtio2021 (Jolokia login: jolokia / jolokia2021) |
| Port | Service |
|---|---|
7443 |
Data Mover — HTTPS portal |
7022 |
Data Mover — SFTP |
8883 |
Data Mover — MQTTS (MQTT over TLS) |
8443 |
Monitor — HTTPS UI |
9640 |
Master — ECpds CLI |
8080 |
Hawtio — authenticated HTTPS JMX console (preset Master, Monitor and Mover connections) |
9404 / 9405 / 9406 |
Prometheus JMX exporter — Master / Mover / Monitor |
Note on FTP: OpenECPDS fully supports FTP in production deployments. However, FTP passive mode (PASV) is not compatible with Docker port mapping — the server advertises its internal container address for data connections, which external clients cannot reach. For that reason, FTP is disabled in this standalone image. Use SFTP (port 7022) as a drop-in alternative for file transfers in Docker.
Note on JMX monitoring: The standalone image includes Hawtio and the Prometheus JMX exporter (details). Point Prometheus/Grafana at
http://localhost:9404/metrics(Master),:9405(Mover) and:9406(Monitor). The credentials above are for evaluation only, and the exporter endpoints are unauthenticated — keep these ports bound to localhost (as in the command above) and never expose them on an untrusted network.
The standalone image ships with a few example destinations (e.g. hourly_aq), but they start out empty and switched off. Before testing the protocols below, log into the Monitoring UI and start one or two of them so there is some data to experiment with. This requires the container to have outbound Internet access, as these destinations acquire data from external providers:
- Open
https://localhost:8443and log in asadmin/admin2021. - On the main page, in the Transmission section, select Destinations.
- Click a destination's name (e.g.
hourly_aq) in the table to open its Data page. - Click the green Start Destination (▶) button above the transfers table.
Once started, the destination retrieves files from its source host and makes them available through the Data Portal, so the commands below have something to list and download.
All examples below use the pre-configured test / test2021 account. The container uses a self-signed certificate — pass the appropriate insecure/skip-verify flag for each tool.
S3 — using AWS CLI:
aws configure set aws_access_key_id test
aws configure set aws_secret_access_key test2021
aws s3 ls s3:// --endpoint-url https://localhost:7443/s3 --no-verify-sslOr with rclone:
rclone lsd :s3: \
--s3-provider=Other \
--s3-endpoint=https://localhost:7443/s3 \
--s3-access-key-id=test \
--s3-secret-access-key=test2021 \
--s3-v2-auth \
--no-check-certificateNote:
--s3-v2-authselects AWS Signature V2 authentication. Use this setting with the rclone example to avoidSignatureDoesNotMatcherrors from incompatible V4 request signing.
WebDAV — using curl:
# List root directory (PROPFIND)
curl -k -u test:test2021 -X PROPFIND https://localhost:7443/webdav/Or with rclone:
rclone lsd :webdav: \
--webdav-url=https://localhost:7443/webdav \
--webdav-vendor=other \
--webdav-user=test \
--webdav-pass=$(rclone obscure test2021) \
--no-check-certificateOr mount directly from your OS file manager:
- macOS Finder: Go → Connect to Server… (⌘K), enter
https://localhost:7443/webdav, click Connect, then log in withtest/test2021. - Windows Explorer: Right-click This PC → Map network drive, choose a drive letter, enter
https://localhost:7443/webdavas the folder, check Connect using different credentials and log in withtest/test2021. - Linux (GNOME Files / Nautilus): Go → Connect to Server…, enter
davs://localhost:7443/webdav, log in withtest/test2021. - Linux (command-line): mount with
davfs2:sudo mount -t davfs https://localhost:7443/webdav /mnt/webdav
Note: All OS file managers will warn about the self-signed certificate — accept it to proceed. The WebDAV share appears as a network drive alongside your local disks.
SFTP — using the sftp command-line client:
sftp -P 7022 -o StrictHostKeyChecking=no test@localhost
# Password: test2021MQTTS — The broker uses TLS with a self-signed certificate.
Using mqttx CLI (recommended — supports true --insecure):
mqttx sub \
-h localhost -p 8883 \
-l mqtts \
-u test -P test2021 \
--insecure \
-t '#'The pre-configured mqtt_demo destination lets you try upload notifications
without an external data provider. Leave this destination stopped in the Monitor:
its standby settings keep uploaded files in HOLD, available for download without
outbound delivery, and trigger MQTT publication.
In one terminal, subscribe with mqttx CLI:
mqttx sub \
-h localhost -p 8883 \
-l mqtts \
-u test -P test2021 \
--insecure \
-t 'openecpds/demo/#'Wait for a successful connection/subscription and leave it running. Messages are
non-retained, so subscribing after publication will not replay them. --insecure
is only for the evaluation container's self-signed certificate.
In a second terminal, create a small file and connect over SFTP:
printf 'Hello from the standalone MQTT demo!\n' > mqtt-demo.txt
sftp -P 7022 test@localhostEnter password test2021, accept the host key after checking it is your evaluation
container, then run at the SFTP prompt:
put mqtt-demo.txt /mqtt_demo/mqtt-demo.txt
bye
Allow roughly a minute for asynchronous publication. The subscriber receives JSON
on openecpds/demo/mqtt-demo.txt, including authenticated download links and, for
files up to 16 KiB, inline Base64 content. Base64 is not encryption.
Verify the file through HTTPS:
curl --fail -k -u test:test2021 \
https://localhost:7443/ecpds/data/file/mqtt_demo/mqtt-demo.txtIn the Monitor, open Operations → MQTT Activity and filter on mqtt_demo.
ACCEPTED means broker acceptance, not proof of subscriber receipt. Use a
different filename for another test upload.
See the full MQTT upload walkthrough
for troubleshooting and public endpoint configuration. For a remote container,
replace localhost in the commands and update the destination's public endpoint
settings so notification links use the reachable hostname and mapped ports.
The ecpds command-line client submits files directly to the Master Server (port 9640).
It is included in the standalone image — no separate installation needed:
docker exec standalone ecpds \
-host localhost -port 9640 \
-user test -pass test2021 \
-dest TEST \
/path/to/myfile.datAlternatively, pull the dedicated CLI image and point it at the container:
# Pull the CLI image
docker pull ghcr.io/ecmwf/open-ecpds/cli:latest
# Submit a file (Linux — use --network host to reach the standalone container)
docker run --rm --network host \
-v /path/to/myfile.dat:/myfile.dat \
ghcr.io/ecmwf/open-ecpds/cli:latest \
-host localhost -port 9640 \
-user test -pass test2021 \
-dest TEST \
/myfile.datOr download a pre-compiled binary from the
GitHub Releases page (ecpds-amd64 /
ecpds-arm64) and run it directly — the binary only requires OpenSSL:
curl -Lo /usr/local/bin/ecpds \
https://github.com/ecmwf/open-ecpds/releases/latest/download/ecpds-amd64
chmod +x /usr/local/bin/ecpds
ecpds -host localhost -port 9640 -user test -pass test2021 -dest TEST /path/to/myfile.datSee Getting the ecpds CLI for the full list of options and installation methods.
# All services (supervisord output)
docker logs -f standalone
# Individual service logs
docker exec standalone tail -f /data/log/master/master.log
docker exec standalone tail -f /data/log/mover/mover.log
docker exec standalone tail -f /data/log/monitor/monitor.logNote: The standalone image runs with
DEBUGlogging enabled by default. This produces verbose output that is useful for exploring how OpenECPDS processes transfers, connects to data movers, and handles incoming requests. In a production deployment the log level can be controlled via theLOG_LEVELenvironment variable (e.g.LOG_LEVEL=warn).
Full documentation is published at ecmwf.github.io/open-ecpds.
| Section | Description |
|---|---|
| Home | Platform overview, core capabilities and quick links |
| Why OpenECPDS | Benefits, domain-agnostic use cases and comparison with Managed File Transfer platforms |
| Getting Started | Requirements, standalone container, installation, first run, IDE setup, ECPDS CLI |
| Architecture | Overview, centralised vs. federated, data ownership & catalogue, components, failover, data transfer lifecycle, continental data movers, data mover allocation & load balancing, database, Java dependencies |
| Deployment | Kubernetes, physical infrastructure, container registry, releasing |
| Concepts | Entities, host types, protocols, plugin infrastructure, object storage, destination/host/web-user options, additional features |
| Monitor UI | Monitoring, destinations, hosts, transfer history, data files, transfer methods & modules, users, administration |
| Administration | Service configuration, TLS certificates, mover control-channel security, TOTP, OpenID Connect, ECaccess identity, critical action password, purge, storage lifecycle policies, product descriptions & status messages, system messages, login page content, destination metadata |
| Service Configuration | Master, Monitor and Data Mover ecmwf.properties reference (under Administration) |
| Use Cases | CLI tool, data portal, data users, web users, data user options, acquisition, acquisition options, dissemination |
| Transfer Modules | ECtrans common options, FTP, FTPS, SFTP, HTTP/S, WebDAV, S3, GCS, Azure Blob, ECauth, optional ECaccess gateways, Portal, Test |
| Host Directory | Acquisition, dissemination, replication, source, backup, proxy scripts |
| Notifications (MQTT) | Real-time dissemination notifications, automated MQTT acquisition, optional WMO WIS2 integration, implementation details |
| Monitoring | Transfer network statistics, TCP socket metrics, JMX with Hawtio & Prometheus, delay notifications |
| Event Logging | PRS, RET, UPH, INH, ERR, CPY, DEA event categories and field reference |
| REST API | REST API reference (v1) |
| API Reference | JavaDocs |
| Global Reach | 1,000+ destinations across 80+ countries |
| Support | Support materials and resources |
| Glossary | Key terms and concepts |
| Changelog | Release notes |
| Contributing | How to contribute |
To build the platform from source and develop against it, you need Docker (with the default socket enabled). Tested on Linux, macOS (Intel/Apple Silicon), and Windows WSL 2.
Run
make helpat any time to list all available targets.
# Build the development container image, start it, and open a shell inside it
make devIf the container already exists, you can reuse it without rebuilding, replace it after a successful build, or cancel. Replacement discards container-only files, not bind-mounted host files.
Alternatively, download a published dev image instead of building it:
make dev-pullRun this on the host from the repository root. It detects the architecture
(override with ARCH=amd64 or ARCH=arm64) and offers to reuse or replace an
existing container, or cancel. This requires a published architecture tag.
See the installation guide
for authentication and runtime credential details.
# Compile Java sources, build RPM packages, and create Docker images
make build
# Start all OpenECPDS services
cd run/bin/ecpds && make upThe monitoring UI is then available at https://localhost:3443 and the data portal at https://localhost:4443.
Local images use the open-ecpds/* namespace. Containers use short names such as
master, mover, monitor and standalone.
Use service keys in Makefile commands, for example make logs svc=master.
Registry images are published under ghcr.io/ecmwf/open-ecpds/*.
For optional local OpenTelemetry testing, run make start-otel from the
repository root, then make logs-otel to inspect decoded logs, traces and
metrics. Use make stop-otel when finished. This diagnostic Collector is not
started by the commands above and is not durable telemetry storage.
See local telemetry setup
for JVM configuration and container networking.
See Getting Started for the full walkthrough, including log inspection, stopping services, IDE setup, and Kubernetes deployment.
# Inside the development container:
make build-saThis stages the RPMs (including the ecpds CLI binary) into docker/ecpds/standalone/ and builds the open-ecpds/standalone:<tag> image locally.
Copyright 2022–2026 ECMWF. Licensed under the Apache License 2.0.