Kubernetes-ready: Fast-Flow runs with Docker (Compose or Socket Proxy) or natively on Kubernetes – pipeline runs as K8s Jobs, without a Docker socket. See k8s/README.md and Kubernetes Deployment.
The lightweight, container-native, Python-centric ETL and task orchestrator for 2026. (Docker Compose or Kubernetes Jobs – configurable via PIPELINE_EXECUTOR.)
Fast-Flow is the answer to the complexity of Airflow and the heaviness of traditional CI/CD tools. It was built for developers who want real isolation without sacrificing the speed of local scripts.
In 60 seconds: One Python script per pipeline, no DAG, no image build. git push → sync → run. Each pipeline runs in isolation with uv (JIT dependencies): under Docker as an ephemeral container (optionally via Socket Proxy), under Kubernetes as a Job pod without Docker on the nodes. One FastAPI orchestrator, ready to go.
(Prefer a scrubbable player? Watch the full-quality MP4.)
Note
Read our Anti-Overhead Manifesto to understand why Fast-Flow is a better alternative to Airflow, Dagster & co.
Tip
Use our fastflow-pipeline-template for a quick start and optimal pipeline structure.
| Dashboard | Pipelines | Pipeline Detail | Dependencies |
|---|---|---|---|
| Dashboard – Overview, metrics, heatmap | Pipelines – List, run, filter | Pipeline Detail – Run history & logs | Dependencies – Packages & CVE check |
- App Overview (Screenshots)
- 🚀 Quick Start (Docker · Local · Kubernetes)
- 🏗 Architecture: The "Runner-Cache" Principle
- 🛠 Worker Process & Lifecycle
- 🔄 Git-Native Deployment
- 🚀 Why Fast-Flow? (Comparison)
- 🎯 Why Fast-Flow Wins (The Python Advantage)
- 🛠 Technical Stack
- 🔒 Security: Docker Socket Proxy
- 📚 Documentation
- 📦 Versioning & Releases
- ❓ Troubleshooting
Get Fast-Flow running in minutes.
- Docker & Docker Compose (for Option 1 and local pipeline runs in Docker mode)
- Python 3.11+ (for local development / key generation only)
- Kubernetes + kubectl (for Option 3; Docker on the workstation is often still needed to build images)
The easiest way to start Fast-Flow.
# 1. Prepare .env file
cp .env.example .env
# 2. Generate encryption key (IMPORTANT!)
# Generates a key and prints it. Add it to .env under ENCRYPTION_KEY.
# For login: GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, INITIAL_ADMIN_EMAIL in .env (see Login section).
python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# 3. Start
docker compose up -d
# 4. View logs
docker compose logs -f orchestratorNote: If your system still uses the legacy CLI, the same commands work with
docker-compose.
Open UI: http://localhost:8000
The orchestrator image starts by default via ./entrypoint.sh (see Dockerfile CMD). The script:
- initializes DB/migrations via
python scripts/init_db_for_migrations.py - in
ENVIRONMENT=development, optionally copies seed pipelines to/app/pipelines - auto-detects
PIPELINES_HOST_DIR(important for Docker worker mounts) - then starts
uvicorn app.main:app --host 0.0.0.0 --port 8000
If you override command or args in Compose/Kubernetes, this init logic should be preserved explicitly (e.g. by calling ./entrypoint.sh).
Uses a local venv but starts containers via Docker.
# 1. Setup
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
# 2. Configuration
cp .env.example .env
# -> Set ENCRYPTION_KEY in .env (see above)
# 3. Start
# Backend (Terminal 1):
uvicorn app.main:app --reload
# Frontend (Terminal 2): cd frontend && npm run devFast-Flow can run on any Kubernetes cluster. Pipeline runs execute as Kubernetes Jobs (no Docker socket required), ideal for production or local K8s setups.
# Apply manifests (adjust Secrets/ConfigMap, see k8s/README.md)
kubectl apply -f k8s/pvc.yaml
kubectl apply -f k8s/postgres.yaml # optional
kubectl apply -f k8s/secrets.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/rbac-kubernetes-executor.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml
# Access: kubectl port-forward svc/fastflow-orchestrator 8000:80 → http://localhost:8000Test locally (Minikube in VM): ./scripts/minikube-vm.sh – see k8s/README.md.
Full guide (images, OAuth, production): k8s/README.md · Kubernetes Deployment (docs).
Authentication is via GitHub, Google, Microsoft (Entra ID), or Custom OAuth (e.g. Keycloak/Auth0):
- GitHub: OAuth app (Settings → Developer settings → OAuth Apps), callback
{BASE_URL}/api/auth/github/callback.
Google: OAuth client (Google Cloud Console), callback{BASE_URL}/api/auth/google/callback.
Microsoft: App registration (Azure/Entra), callback{BASE_URL}/api/auth/microsoft/callback.
Custom OAuth: Callback{BASE_URL}/api/auth/custom/callbackplus provider endpoints (AUTHORIZE_URL,TOKEN_URL,USERINFO_URL). - In
.env: Set credentials for at least one provider (GITHUB_*,GOOGLE_*,MICROSOFT_*, orCUSTOM_OAUTH_*) andINITIAL_ADMIN_EMAIL(email for the first admin). - Docker (everything on :8000): omit
FRONTEND_URLor set=http://localhost:8000,BASE_URL=http://localhost:8000.
Dev (frontend :3000, backend :8000):FRONTEND_URL=http://localhost:3000,BASE_URL=http://localhost:8000.
Tip
Detailed steps, invitations, account linking, join requests: OAuth (GitHub, Google, Microsoft, Custom).
Join requests (knock-to-join): Unknown users (without an invitation) can submit a request via OAuth. They receive no session, are redirected to /request-sent, and appear under Users → Join Requests. After approval by an admin, they can log in normally. Rejected or still-pending users land on /request-sent or /request-rejected on subsequent OAuth login (also without a session).
Unlike classic orchestrators that often suffer from "dependency hell" in their worker environments, Fast-Flow uses a modern JIT environment architecture.
Via PIPELINE_EXECUTOR you choose the backend: docker (Compose / Docker API) or kubernetes (Batch API → Jobs). Detailed overview: Architecture (docs).
- The Singleton Brain: A single FastAPI process manages state, the scheduler, and Git sync.
- Ephemeral Workers: Each pipeline starts in an isolated sandbox – Docker container or Kubernetes Job pod. No side effects between runs.
- uv-Acceleration: The global uv cache and uv Python installations (e.g. 3.11, 3.12) are persistent (host volumes or PVCs) and mounted into the worker. Dependencies and the per-pipeline selectable Python version (from
pipeline.jsonorDEFAULT_PYTHON_VERSION) are available in milliseconds – without a fixed Python in the base image. - Live-Streaming: Logs and metrics (CPU/RAM) via SSE: under Docker via container logs and Docker stats (through socket proxy); under Kubernetes via pod logs and optionally the Metrics API (metrics-server).
Fast-Flow uses a "Disposable Worker" model. For each execution, a fresh, isolated container (Docker) or pod (Kubernetes Job) is created. The flow:
When a run is triggered via the React frontend (manually) or APScheduler (scheduled):
- The API validates the pipeline structure and loads encrypted secrets.
- A new entry is created in the SQLite database with status
PENDING.
This is the core of Fast-Flow performance. Instead of building a pipeline-specific image, a generic worker image starts (uv, optionally pre-installed Python):
- Docker: Pipeline directory mounted from the host (read-write, so pipelines can write output files), uv cache and uv Python mounted.
- Kubernetes: The orchestrator copies the pipeline into a shared volume before the Job (
pipeline_runs/<Run-ID>on the cache PVC); Jobs mount uv cache and uv Python the same way as in the K8s docs. - Just-In-Time (both):
uv run --python {version}– version per pipeline (python_versionin pipeline.json) orDEFAULT_PYTHON_VERSION. Python fromuv python install(preheating).- Dependencies in cache? → Linked in milliseconds via hardlink.
- New dependencies? → Loaded once, stored in the shared cache for future runs.
- Preheating: On startup and after Git sync:
uv python install/uv pip compile– low latency on first run.
While the worker runs, FastAPI handles:
- Logs: stdout/stderr asynchronously → SSE (Docker: container logs; Kubernetes: pod logs).
- Metrics: Docker stats through the proxy or Kubernetes Metrics API for the run pod (when available).
- Security: With Docker, no direct socket access – only via Docker Socket Proxy. With Kubernetes, the orchestrator talks to the Kubernetes API (RBAC/ServiceAccount), no host Docker socket for runs.
After the Python script completes:
- The exit code is captured (e.g. 137 for OOM errors).
- Docker: Container is removed (
--rm). Kubernetes: Job ends; TTL/cluster policy cleans up resources; pipeline copy in the volume is cleaned up. - Logs are finalized and stored for long-term archival.
flowchart TB
subgraph ClientLayer["Client Layer"]
A["🌐 React Frontend<br/><small>TypeScript + Vite</small>"]
end
subgraph AppLayer["Application Layer"]
B["⚡ FastAPI Orchestrator<br/><small>Python 3.11+</small>"]
C[("💾 Database<br/><small>SQLite/PostgreSQL</small>")]
D["🔐 Auth & Secrets<br/><small>OAuth / Fernet</small>"]
end
A -->|"REST/SSE Live Updates"| B
B -->|"SQLModel ORM"| C
B -->|"JWT & Encryption"| D
PE{"PIPELINE_EXECUTOR"}
B --> PE
subgraph DockerPfad["Docker (e.g. Compose)"]
E["🛡️ Docker Socket Proxy<br/><small>tecnativa</small>"]
F["🐳 Docker Daemon"]
G["📦 Pipeline Container<br/><small>uv run --python</small>"]
H["📚 uv Cache"]
J["🐍 uv Python"]
I["📝 Pipeline Code<br/><small>rw mount</small>"]
end
subgraph K8sPfad["Kubernetes Jobs"]
K["☸️ Kubernetes API<br/><small>BatchV1 Jobs</small>"]
L["📦 Job Pod<br/><small>uv run --python</small>"]
M["📚 PVC: uv + pipeline_runs"]
end
PE -->|docker| E
E --> F
F --> G
G -.-> H
G -.-> J
G -.-> I
B -.->|"HTTP to Proxy"| E
B -.->|"Logs & Stats"| G
PE -->|kubernetes| K
K --> L
L -.-> M
B -.->|"RBAC / SA"| K
B -.->|"Pod Logs & Metrics"| L
classDef frontend fill:#61dafb,stroke:#20232a,stroke-width:2px,color:#000
classDef backend fill:#009688,stroke:#004d40,stroke-width:2px,color:#fff
classDef security fill:#ff9800,stroke:#e65100,stroke-width:2px,color:#000
classDef infra fill:#2196f3,stroke:#0d47a1,stroke-width:2px,color:#fff
classDef execution fill:#9c27b0,stroke:#4a148c,stroke-width:2px,color:#fff
classDef storage fill:#607d8b,stroke:#263238,stroke-width:2px,color:#fff
classDef decision fill:#fff9c4,stroke:#f57f17,stroke-width:2px,color:#000
class A frontend
class B,C,D backend
class E security
class F,K infra
class G,L execution
class H,I,J,M storage
class PE decision
- Speed: Without
docker buildsteps, a pipeline starts as fast as a local process. - Isolation: A failure in
pipeline_acan never affect the environment ofpipeline_b. - Scalability: Controller (API) and workers (containers or Jobs) are decoupled; future extensions e.g. with message queues (Redis) are possible.
Push to Deploy, No Build Needed
Traditional orchestrators often turn deployment into a logistics problem. Fast-Flow turns it into a git push.
- Image Hell: Every code change often requires a new Docker build (wait 5–10 minutes).
- Sidecar Chaos: You need complex Git-sync sidecars or S3 buckets to distribute DAGs.
- Version Gap: What you see in the UI often doesn't match what's in the Git repository.
In Fast-Flow, your Git repository is the single source of truth. There is no "Upload" button and no manual build step.
- Zero-Build Deployment: When you change your code, the orchestrator pulls changes via auto-sync or manual sync. Thanks to the uv JIT architecture, the new version is ready to run immediately.
- Full Traceability: Since every pipeline configuration (
pipeline.json) and every library (requirements.txt) lives in Git, you have a complete history. Who increased the memory limit when? Who changed the prophet version? Your Git log tells you. - Atomic Sync: Our sync mechanism ensures pipelines never read "half" files. Changes are applied atomically – safe and consistent.
| Feature | Traditional Tools | Fast-Flow |
|---|---|---|
| Deployment Speed | Minutes (build & push) | Seconds (git pull) |
| Versioning | Often code only | Code, deps & resource limits |
| Rollback | Image rollback (complex) | Git revert (simple) |
| Source of Truth | UI vs. Git vs. image | Git is law |
- Develop: Write your Python script locally.
- Push:
git push origin main. - Sync: The orchestrator detects the change (via auto-sync or manual sync).
- Run: The pipeline starts immediately with the new code. No Docker builds, no waiting.
"We made deployment as boring as possible so you can focus on the exciting part: your code."
| Feature | Fast-Flow | Airflow | Dagster | Prefect | Kestra |
|---|---|---|---|---|---|
| Pipeline definition | 🟢 Plain Python script (main.py) |
🔴 DAGs, operators, XComs | 🟡 Assets, ops, resources | 🟡 Python + @flow/@task decorators |
🟡 Declarative YAML |
| Setup | 🟢 Compose or K8s manifests | 🔴 Complex cluster | 🟡 Medium | 🟡 Server + workers (or Cloud) | 🟡 Standalone service (split for HA) |
| Isolation | 🟢 Strict (container/job per run) | 🔴 Weak (shared worker) | 🟡 Medium | 🟡 Depends on worker type | 🟡 Opt-in per task (Docker/K8s runner) |
| Dependency Speed | 🟢 Instant (uv JIT, no image build) | 🔴 Slow (image builds) | 🟡 Medium | 🟡 Env-dependent | 🟡 Plugin/image based |
| Deployment | 🟢 Git push + auto-sync | 🔴 Complex CI/CD | 🟡 Code deployment | 🟡 Code deployment | 🟡 Push / API |
| UI | 🟢 Modern & realtime (React) | 🔴 Dated / static | 🟢 Modern | 🟢 Modern | 🟢 Modern |
| Onboarding | 🟢 Minutes — if it runs locally, it runs here | 🔴 Weeks | 🟡 Days | 🟡 Days | 🟡 Days |
Fast-Flow is deliberately narrow: "I have a handful of Python scripts and Airflow is overkill." If that's you, the whole point is that there's no DAG, no decorators, no image build — you push a main.py and it runs in an isolated container. That's the sweet spot.
It's not trying to be everything, and each of these tools has a domain it owns:
- Kestra — a large plugin catalog and native event-driven triggers (files, webhooks, Kafka, queues) for wiring many systems together.
- Dagster — data-asset lineage as its core model, when you need to track how datasets depend on and produce one another.
- Prefect — a richer Python-first orchestration engine (retries, state, flexible workers) when a single
main.pyisn't enough but you still want to stay in Python.
Fast-Flow trades that breadth for a radically simpler mental model and near-zero operational overhead. If your problem is "a handful of Python scripts and Airflow is overkill," that trade is the whole point.
In other orchestrators you often have to write YAML files or wrestle with complex DSLs.
- The pipelines: A pipeline is a simple Python script. If it runs locally, it runs in the orchestrator. No special decorators, no operator classes, no complex configuration.
- The frontend: Modern React dashboard with real-time monitoring. The backend stays 100% Python (FastAPI).
No proprietary logic: You don't need to learn special decorators (like @dag) or operators (PythonOperator).
- "Write & Run": New developers can push their first pipeline within 5 minutes. If you understand Python, you understand Fast-Flow.
- Local debugging: Since we use uv, developers can reproduce the exact same environment locally with one command that also runs in the container.
Onboarding with Airflow: Often a matter of days or weeks (due to DSL, providers, cluster logic) – with Fast-Flow it's a matter of minutes.
While Airflow needs a Postgres DB, a Redis broker, a scheduler, a webserver, and multiple workers, Fast-Flow stays deliberately lean: typically one orchestrator deployment plus ephemeral workers (Docker containers or K8s Jobs).
- Low maintenance: Update e.g.
docker compose pullor deploy a new orchestrator image in the cluster. - Resource-efficient: Ideal for edge servers or smaller VM instances.
- Zero-Build Pipelines: You don't need to build Docker images for your pipelines. Change requirements.txt in Git, and Fast-Flow warms the cache automatically in the background.
- No "Database is locked": Optimized for SQLite with WAL mode and async I/O.
- Resource Control: Set CPU and RAM limits per pipeline directly via JSON metadata.
- Security Focus: Encrypted secrets (Fernet) and native GitHub App support.
- Backend: FastAPI (Python 3.11+)
- Frontend: React + TypeScript (Vite)
- Database: SQLModel (SQLite/PostgreSQL) – Production: PostgreSQL recommended
- Execution: Docker Engine API + uv or Kubernetes Jobs (K8s-ready, see k8s/README.md)
- Security: Docker Socket Proxy (tecnativa/docker-socket-proxy) in Docker mode; no socket exposure needed on K8s
- Scheduling: APScheduler (persistent)
- Auth: OAuth (GitHub, Google, Microsoft, Custom), JWT & Fernet encryption
- Automatic Pipeline Discovery: Pipelines are automatically discovered from a Git repository
- Isolated Execution: Each pipeline in its own Docker container or Kubernetes Job (
PIPELINE_EXECUTOR) - Resource Management: Configurable CPU and memory limits per pipeline
- Scheduling: Support for CRON- and interval-based jobs
- Webhooks: Trigger pipelines via HTTP webhooks
- Live Monitoring: Real-time logs and metrics during execution
- Git Sync: Automatic synchronization with Git repositories
- Secrets Management: Secure management of secrets and parameters
- S3 Log Backup (optional): Pipeline logs are backed up to S3/MinIO before local deletion (cleanup); deletion only happens after successful upload. On failure: UI notice and email to
EMAIL_RECIPIENTS. See Log Backup (S3/MinIO).
With PIPELINE_EXECUTOR=docker (default with Docker Compose), Fast-Flow uses a Docker Socket Proxy (tecnativa/docker-socket-proxy) between the orchestrator and the Docker daemon. This eliminates direct root access to the socket; only selected Docker API operations are allowed. With PIPELINE_EXECUTOR=kubernetes, the orchestrator talks to the Kubernetes API (Jobs/Pods); the proxy is not needed.
- Security: The Docker socket (
/var/run/docker.sock) effectively grants root access to the entire host system. A proxy filters and allows only configured operations. - Controlled Access: Only container creation, logs, stats, and image pulls are allowed. Network and volume management are disabled.
- Isolation: Even if the orchestrator is compromised, damage is limited.
The proxy is configured automatically in docker-compose.yaml:
docker-proxy:
image: tecnativa/docker-socket-proxy:latest
environment:
- CONTAINERS=1 # Allow container operations
- IMAGES=1 # Allow image pulls
- BUILD=1 # Allow build operations
- VOLUMES=1 # Allow volume mounts
- EXEC=1 # Allow exec (used to read the uv version from the container)
- POST=1 # Allow HTTP POST (container creation)
- DELETE=1 # Allow container removal
- STATS=1 # Allow resource monitoring
- NETWORKS=0 # Disable network management
- SYSTEM=0 # Disable system operationsThe orchestrator communicates with the proxy via http://docker-proxy:2375 instead of directly with the Docker socket.
Documentation lives under docs/docs/ and is served with Docusaurus. Start locally: cd docs && npm run start → http://localhost:3000/docs.
| Area | Links |
|---|---|
| Getting Started | Quick Start · Setup Guide · Manifesto · Architecture |
| Pipelines | Overview · First Pipeline · Advanced Pipelines · pipeline.json Reference |
| Operations | Configuration · Production · Git Deployment · Kubernetes · Docker Socket Proxy |
| Security & Ops | OAuth (GitHub, Google, Microsoft, Custom) · S3 Log Backup · Compliance |
| Reference | API · Database/Schema · Versioning |
| Help | Troubleshooting · Disclaimer |
Fast-Flow uses an automated version check that runs daily to see if new releases are available.
The version is stored in the VERSION file in the project root (currently e.g. v1.0.4):
v1.0.4
To publish a new version:
-
Update VERSION file:
export NEW_VERSION=v1.0.5 echo "$NEW_VERSION" > VERSION git add VERSION git commit -m "Bump version to $NEW_VERSION"
-
Create tag (must match VERSION file exactly):
git tag "$NEW_VERSION" git push origin "$NEW_VERSION"
-
Create GitHub release:
- Go to: https://github.com/ttuhin03/fastflow/releases/new
- Select tag:
$NEW_VERSION(e.g.v1.0.5) - Add release notes
- Publish the release
Important: The tag format must exactly match the VERSION file (both with "v" prefix)
The version check runs automatically:
- ✅ On API startup
- ✅ Daily at 2:00 AM (together with log cleanup)
- ✅ On demand via API:
GET /api/system/version?force_check=true
More details: Versioning & Releases
The pipeline repository lives under PIPELINES_DIR – locally, via volume in the orchestrator (Docker Compose), or on a PVC (Kubernetes). Pipelines are discovered automatically; in K8s Jobs mode, the orchestrator copies the snapshot per run to the cache volume.
Tip
Use our fastflow-pipeline-template for a quick start and optimal pipeline structure.
pipelines/
├── pipeline_a/
│ ├── main.py # Main pipeline script (required)
│ ├── requirements.txt # Python dependencies (optional)
│ └── pipeline.json # Metadata (optional)
├── pipeline_b/
│ ├── main.py
│ ├── requirements.txt
│ └── data_processor.json # Alternative: {pipeline_name}.json
└── pipeline_c/
└── main.py # Minimal: main.py only
The main pipeline script. Every pipeline must have a main.py file in its own directory.
Execution:
- Pipelines are run with
uv run --with-requirements {requirements.txt} {main.py} - Code can run top to bottom (no
main()function required) - Optional:
main()function withif __name__ == "__main__"block
Example 1: Simple script (top to bottom)
# main.py
import os
print("Pipeline started")
data = os.getenv("MY_SECRET")
print(f"Processing data: {data}")
# ... more code ...Example 2: With main() function (optional)
# main.py
def main():
print("Pipeline started")
# ... logic ...
if __name__ == "__main__":
main()Error handling:
- Uncaught exceptions cause Python to return exit code != 0 automatically
- Pipeline is marked as
FAILED
Python dependencies for the pipeline. Installed dynamically by uv.
Format: Standard Python requirements.txt format
requests==2.31.0
pandas==2.1.0
numpy==1.24.3
Notes:
- Dependencies are installed automatically on pipeline start (via
uv) - Shared cache enables fast installation (< 1 second with cached dependencies)
- Pre-heating: Dependencies can be preloaded on Git sync (UV_PRE_HEAT)
Metadata file for resource limits and configuration.
Filenames:
pipeline.json(default, preferred){pipeline_name}.json(alternative, e.g.data_processor.json)
JSON format:
{
"cpu_hard_limit": 1.0,
"mem_hard_limit": "1g",
"cpu_soft_limit": 0.8,
"mem_soft_limit": "800m",
"timeout": 3600,
"retry_attempts": 3,
"description": "Processes incoming data daily",
"tags": ["data-processing", "daily"],
"enabled": true,
"default_env": {
"LOG_LEVEL": "INFO",
"DEBUG": "false"
}
}Fields:
Resource limits:
cpu_hard_limit(Float, optional): CPU limit in cores (e.g. 1.0 = 1 core, 0.5 = half core)mem_hard_limit(String, optional): Memory limit (e.g. "512m", "1g", "2g")cpu_soft_limit(Float, optional): CPU soft limit for monitoring (monitored only, not enforced)mem_soft_limit(String, optional): Memory soft limit for monitoring (monitored only, not enforced)
Pipeline configuration:
timeout(Integer, optional): Timeout in seconds (pipeline-specific, overrides global CONTAINER_TIMEOUT)retry_attempts(Integer, optional): Number of retry attempts on failure (pipeline-specific, overrides global RETRY_ATTEMPTS)enabled(Boolean, optional): Pipeline enabled/disabled (default: true)
Documentation:
description(String, optional): Pipeline description (shown in UI)tags(Array[String], optional): Tags for categorization/filtering in the UI
Environment variables:
default_env(Object, optional): Pipeline-specific default environment variables- Set on every pipeline start
- Can be supplemented with additional env vars in the UI (merged)
- Useful for pipeline-specific configuration (e.g. LOG_LEVEL, API_ENDPOINT, etc.)
- Secrets should NOT be stored here (use secrets management in the UI instead)
Behavior:
- Hard limits: Set when the worker starts (Docker cgroups or Kubernetes
resources.limits)- Exceeding memory leads to OOM kill (exit code 137)
- CPU is throttled when exceeded
- Soft limits: Monitored only, not enforced
- Exceeding is shown in the frontend (warning)
- Useful for early detection of resource issues
- Missing metadata: Default limits are used (if configured)
- Timeout & retry: Pipeline-specific values override global configuration
- Environment variables:
default_envis merged with UI-specific env vars (UI values take precedence)
Example:
{
"cpu_hard_limit": 2.0,
"mem_hard_limit": "2g",
"cpu_soft_limit": 1.5,
"mem_soft_limit": "1.5g"
}- Automatic discovery: Pipelines are discovered automatically on Git sync
- Pipeline name: Matches the directory name (e.g.
pipeline_a/→ pipeline name:pipeline_a) - Validation: Pipeline must contain a
main.pyfile, otherwise it is ignored - No manual registration: Pipelines become available automatically
Complete example:
pipelines/
└── data_processor/
├── main.py
├── requirements.txt
└── data_processor.json
main.py:
import os
import requests
import json
def process_data():
api_key = os.getenv("API_KEY")
data = fetch_data(api_key)
result = transform_data(data)
save_result(result)
def fetch_data(api_key):
response = requests.get("https://api.example.com/data", headers={"Authorization": f"Bearer {api_key}"})
return response.json()
def transform_data(data):
# ... transformation logic ...
return data
def save_result(result):
with open("/tmp/result.json", "w") as f:
json.dump(result, f)
if __name__ == "__main__":
process_data()requirements.txt:
requests==2.31.0
data_processor.json:
{
"cpu_hard_limit": 1.0,
"mem_hard_limit": "512m",
"cpu_soft_limit": 0.8,
"mem_soft_limit": "400m",
"timeout": 1800,
"retry_attempts": 2,
"description": "Processes incoming data and creates reports",
"tags": ["data-processing", "reports"],
"enabled": true,
"default_env": {
"LOG_LEVEL": "INFO",
"API_ENDPOINT": "https://api.example.com"
}
}More docs: Pipelines – Overview, Configuration
Make sure Docker Desktop is running.
Check: docker ps
The orchestrator may only execute certain commands. Check proxy logs:
docker compose logs docker-proxy
Make sure POST=1 (for container start) is set.
Change PORT in the .env file.
The application won't start without a key. Generate one (see Quick Start) and set it in .env.
Important notice on security and liability:
This project is in an early stage / beta. In PIPELINE_EXECUTOR=docker mode, the orchestrator has indirect access to the Docker daemon (via the recommended proxy) – misconfiguration poses a relevant risk to the host system. Under kubernetes, runs are decoupled from host Docker; typical K8s concerns apply instead (RBAC, networking, secrets).
- Use at your own risk: The software is provided "as is". The author assumes no liability for damage to hardware, data loss, security vulnerabilities, or service interruptions that may result from use of this software.
- No warranty: There is no guarantee of correctness, functionality, or continuous availability of the software.
- Security recommendation: Never run unprotected on the public internet. In Docker mode, use the socket proxy and strong authentication; in Kubernetes, grant minimal RBAC rights to the orchestrator ServiceAccount.
Full details in the docs: Disclaimer.