ETLR

Command-line interface for deploying and managing ETLR workflows.

Quick reference: See QUICKSTART.md for command cheat sheet.

Installation

pip install etlr

Authentication

Get Your API Key

Get your API key from the ETLR dashboard: https://app.etlr.io/developer

Recommended: Environment Variable

Set your API key once in your shell configuration:

# Add to ~/.zshrc or ~/.bashrc
export ETLR_API_KEY=your_api_key_here

Reload your shell or run:

source ~/.zshrc  # or ~/.bashrc

Alternative: CLI Flag

Override or specify per-command:

etlr --api-key your_api_key_here list

Priority: CLI flag > ETLR_API_KEY environment variable

Quick Start

1. Create a workflow

# workflow.yaml
name: hello-world
stage: dev

input:
  type: http_webhook

steps:
  - type: print
    message: "Hello from ${input.data}!"

2. Deploy it

etlr deploy workflow.yaml

Output:

Pushing workflow...
✓ Workflow created: hello-world/dev
Deploying workflow...
✓ Workflow deployed and running

3. Check status

etlr status --name hello-world --stage dev

4. List all workflows

etlr list

That's it! Your workflow is running.

Environment Variables

Declare environment variables in your workflow YAML, and the CLI automatically gathers them from your shell environment.

Basic Usage

1. Declare in workflow.yaml:

workflow:
  name: data-processor
  stage: dev

  environment:
    - name: API_KEY
      secret: true
    - name: DATABASE_URL
      secret: true
    - name: LOG_LEVEL

  input:
    type: http_webhook

  steps:
    - type: http_call
      url: https://api.example.com/data
      headers:
        Authorization: "Bearer ${env:API_KEY}"

2. Set values in your shell:

export API_KEY=sk-abc123...
export DATABASE_URL=postgres://...
export LOG_LEVEL=info

3. Deploy:

etlr deploy workflow.yaml

Output:

Environment variables:
  API_KEY: *** (secret)
  DATABASE_URL: *** (secret)
  LOG_LEVEL: info

Pushing workflow...
✓ Workflow deployed

Secret Masking

Mark sensitive values as secret: true to hide them in CLI output:

environment:
  - name: API_KEY
    secret: true       # Shows as ***
  - name: LOG_LEVEL
    secret: false      # Shows actual value
  - name: TIMEOUT      # Defaults to false

CLI Overrides

Override declared env vars or add new ones with -e flag:

# Override
etlr deploy workflow.yaml -e API_KEY=different-key

# Add new variable
etlr deploy workflow.yaml -e EXTRA_VAR=value

# Multiple overrides
etlr deploy workflow.yaml -e API_KEY=xxx -e LOG_LEVEL=debug

Missing Variables

If required env vars are missing, deployment fails with a helpful error:

$ etlr deploy workflow.yaml
Error: Missing required environment variables: API_KEY, DATABASE_URL

Set them with:
  export API_KEY=value
  export DATABASE_URL=value

Or use -e flags:
  etlr deploy workflow.yaml -e API_KEY=value -e DATABASE_URL=value

Benefits

✅ Self-documenting - Anyone reading YAML knows what's needed
✅ Validated - Missing vars caught before deployment
✅ No CLI clutter - No need for multiple -e flags
✅ Version controlled - Variable names (not values) in git
✅ Team friendly - New developers see requirements immediately

Stage Management

Stages (dev, staging, prod) let you run different versions of the same workflow in different environments.

Four Ways to Set Stage

Priority (highest to lowest):

--stage CLI flag (explicit override)
ETLR_STAGE environment variable (session default)
stage field in YAML (file default)
${env:STAGE} in YAML (dynamic from environment)

Method 1: Static in YAML (Simple)

name: my-workflow
stage: dev

etlr deploy workflow.yaml           # Uses 'dev'
etlr deploy workflow.yaml --stage prod  # Override to 'prod'

Use when: Single environment or simple setup

Method 2: ETLR_STAGE Variable (Recommended)

# Set once for your session
export ETLR_STAGE=dev

# Deploy without specifying stage
etlr deploy workflow.yaml           # Uses 'dev'

# Override when needed
etlr deploy workflow.yaml --stage prod

Add to ~/.zshrc for persistence:

export ETLR_STAGE=dev
export ETLR_API_KEY=your_dev_key

Use when: Local development with occasional prod deploys

Method 3: Dynamic in YAML (Flexible)

name: my-workflow
stage: ${env:STAGE}     # or ${env:STAGE, dev} with default

STAGE=dev etlr deploy workflow.yaml
STAGE=prod etlr deploy workflow.yaml

Use when: CI/CD or multiple environments

Method 4: CLI Flag (Most Explicit)

etlr deploy workflow.yaml --stage staging
etlr deploy workflow.yaml --stage prod

Use when: CI/CD pipelines or when you want explicit control

Best Practices by Environment

Local Development:

# ~/.zshrc
export ETLR_STAGE=dev
export ETLR_API_KEY=dev_key

# Just deploy
cd my-project
etlr deploy workflow.yaml

CI/CD:

# .github/workflows/deploy.yml
- name: Deploy to staging
  run: etlr deploy workflow.yaml --stage staging
  env:
    ETLR_API_KEY: ${{ secrets.STAGING_API_KEY }}

- name: Deploy to production
  run: etlr deploy workflow.yaml --stage prod
  env:
    ETLR_API_KEY: ${{ secrets.PROD_API_KEY }}

Team with .env files:

# .env.dev
ETLR_STAGE=dev
ETLR_API_KEY=dev_key

# .env.prod
ETLR_STAGE=prod
ETLR_API_KEY=prod_key

# Deploy
source .env.dev && etlr deploy workflow.yaml
source .env.prod && etlr deploy workflow.yaml

Commands

List Workflows

etlr list

Deploy Workflow

Creates/updates and starts a workflow in one command.

# From file
etlr deploy workflow.yaml

# From workflow.yaml in current directory
etlr deploy

# Override stage
etlr deploy workflow.yaml --stage prod

# With environment variables
etlr deploy workflow.yaml -e API_KEY=xxx -e LOG_LEVEL=debug

# By identifier (if already pushed)
etlr deploy --id <workflow-uuid>
etlr deploy --name my-workflow --stage prod

Get Workflow

# By ID
etlr get --id <workflow-uuid>

# By name and stage
etlr get --name my-workflow --stage prod

Start Workflow

Start a workflow that was previously stopped or pushed but not started.

etlr start --id <workflow-uuid>
etlr start --name my-workflow --stage prod

Stop Workflow

etlr stop --id <workflow-uuid>
etlr stop --name my-workflow --stage prod

Get Status

etlr status --id <workflow-uuid>
etlr status --name my-workflow --stage prod

Output shows health status with color coding:

🟢 Green: Running normally
🟡 Yellow: Paused or warning
🔴 Red: Error or stopped

View Logs

View workflow execution logs to debug issues or monitor activity.

# View recent logs (last 100 lines by default)
etlr logs --name my-workflow --stage prod
etlr logs --id <workflow-uuid>

# Specify number of lines
etlr logs --name my-workflow --stage prod --lines 200
etlr logs --id <workflow-uuid> -n 500

# Raw output (no formatting)
etlr logs --name my-workflow --stage prod --raw

Output features:

Color-coded log levels (ERROR=red, WARN=yellow, INFO=blue, DEBUG=white)
Timestamps for each log entry
Formatted display showing level and message
Raw mode for piping to other tools

Example output:

=== Logs for my-workflow/prod (last 100 lines) ===

2024-01-15T10:00:00Z [INFO] Workflow started
2024-01-15T10:00:01Z [DEBUG] Processing event data
2024-01-15T10:00:02Z [INFO] HTTP request completed successfully
2024-01-15T10:00:03Z [WARN] Rate limit approaching
2024-01-15T10:00:04Z [ERROR] Failed to connect to database

Delete Workflow

# With confirmation prompt
etlr delete --id <workflow-uuid>
etlr delete --name my-workflow --stage prod

# Skip confirmation (for scripts)
etlr delete --name my-workflow --stage prod --yes

Version Management

ETLR automatically versions workflows on each deploy. You can list, view, and restore previous versions.

List Versions

etlr versions --id <workflow-uuid>

Output:

Versions for workflow abc-123:
  Version 3 (current) - 2025-12-17 14:23:10
  Version 2 - 2025-12-17 12:15:43
  Version 1 - 2025-12-16 09:30:22

Get Specific Version

etlr get-version --id <workflow-uuid> --version 2

Restore Previous Version

# With confirmation
etlr restore --id <workflow-uuid> --version 2

# Skip confirmation
etlr restore --id <workflow-uuid> --version 2 --yes

This creates a new version (e.g., version 4) with the content from version 2.

CI/CD Integration

GitHub Actions

name: Deploy Workflow

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install ETLR CLI
        run: pip install etlr

      - name: Deploy to staging
        if: github.ref == 'refs/heads/main'
        run: etlr deploy workflow.yaml --stage staging
        env:
          ETLR_API_KEY: ${{ secrets.STAGING_API_KEY }}

      - name: Deploy to production
        if: github.ref == 'refs/heads/main' && github.event_name == 'release'
        run: etlr deploy workflow.yaml --stage prod
        env:
          ETLR_API_KEY: ${{ secrets.PROD_API_KEY }}

GitLab CI

stages:
  - deploy

deploy_staging:
  stage: deploy
  script:
    - pip install etlr
    - etlr deploy workflow.yaml --stage staging
  environment:
    name: staging
  variables:
    ETLR_API_KEY: $STAGING_API_KEY

deploy_production:
  stage: deploy
  script:
    - pip install etlr
    - etlr deploy workflow.yaml --stage prod
  environment:
    name: production
  variables:
    ETLR_API_KEY: $PROD_API_KEY
  when: manual

Tips for CI/CD

Store API keys in secret managers (ETLR_API_KEY)
Use --stage flag for explicit environment targeting
Use --yes flag to skip confirmations
Set timeouts for deployment commands
Add status checks after deployment

Development

Setup

git clone https://github.com/etlr-io/cli.git
cd cli

python3 -m venv .venv
source .venv/bin/activate

pip install -e ".[dev]"

Run

etlr --help

Testing

pytest
pytest -v  # Verbose
pytest tests/test_cli.py  # Specific file

Code Quality

# Format
black src tests

# Lint
ruff check src tests

# Type check
mypy src

Release

# Update version in pyproject.toml
# Commit and tag
git tag v1.2.3
git push origin v1.2.3

# Build and publish
python -m build
twine upload dist/*

Tips & Tricks

Per-Project Configuration

Use .env files or direnv for project-specific settings:

# .env
ETLR_API_KEY=project_specific_key
ETLR_STAGE=dev

# Load and deploy
source .env
etlr deploy workflow.yaml

Or use direnv to auto-load when entering directory:

# .envrc
export ETLR_API_KEY=project_key
export ETLR_STAGE=dev

Multiple Accounts

Switch between accounts with CLI flag:

# Production account
etlr --api-key $PROD_KEY deploy workflow.yaml

# Staging account
etlr --api-key $STAGING_KEY deploy workflow.yaml

Workflow Aliases

Create shell aliases for common workflows:

# ~/.zshrc
alias deploy-prod='etlr deploy workflow.yaml --stage prod'
alias deploy-dev='etlr deploy workflow.yaml --stage dev'
alias check-prod='etlr status --name my-workflow --stage prod'

Debugging

Enable verbose output (when available):

etlr --debug deploy workflow.yaml

Or check workflow logs via the web dashboard.

Getting Help

Quick reference: QUICKSTART.md
Issues: https://github.com/etlr-io/cli/issues
Documentation: https://etlr.io/docs/

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
.github/workflows		.github/workflows
src/etlr		src/etlr
tests		tests
tools		tools
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
QUICKSTART.md		QUICKSTART.md
README.md		README.md
pyproject.toml		pyproject.toml

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

ETLR

Table of Contents

Installation

Authentication

Get Your API Key

Recommended: Environment Variable

Alternative: CLI Flag

Quick Start

1. Create a workflow

2. Deploy it

3. Check status

4. List all workflows

Environment Variables

Basic Usage

Secret Masking

CLI Overrides

Missing Variables

Benefits

Stage Management

Four Ways to Set Stage

Method 1: Static in YAML (Simple)

Method 2: ETLR_STAGE Variable (Recommended)

Method 3: Dynamic in YAML (Flexible)

Method 4: CLI Flag (Most Explicit)

Best Practices by Environment

Commands

List Workflows

Deploy Workflow

Get Workflow

Start Workflow

Stop Workflow

Get Status

View Logs

Delete Workflow

Version Management

List Versions

Get Specific Version

Restore Previous Version

CI/CD Integration

GitHub Actions

GitLab CI

Tips for CI/CD

Development

Setup

Run

Testing

Code Quality

Release

Tips & Tricks

Per-Project Configuration

Multiple Accounts

Workflow Aliases

Debugging

Getting Help

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages