A high-performance, redis-backed distributed task queue system that lets you write workers in any language via WebAssembly. Features priority queuing, task scheduling, dead letter queues, and real-time event monitoring.
- Language Agnostic: Write workers in any language that compiles to WebAssembly
- Flexible Queue Types: FIFO, LIFO, Priority, Delayed, and Composite queues
- Advanced Task Management:
- Task dependencies and DAG support
- Scheduled and periodic tasks
- Batch processing capabilities
- Task versioning and migration
- Robust Architecture:
- Redis-backed persistence
- Distributed task processing
- Auto-scaling worker pools
- Dead letter queue handling
- Real-time Monitoring:
- Live queue metrics
- Worker performance tracking
- Task execution statistics
- Distributed tracing
- Quick Start
- Architecture
- Use Cases
- Installation
- Configuration
- Examples
- Documentation
- Contributing
- License
# Still in DEVOtterMQ is built on a modular architecture that prioritizes scalability and flexibility:
---
title: OtterMQ High-Level Architecture
---
graph TB
%% Node Definitions
classDef client fill:#87CEFA,stroke:#1E90FF,stroke-width:2px;
classDef api fill:#FFD700,stroke:#FF8C00,stroke-width:2px;
classDef taskManager fill:#98FB98,stroke:#32CD32,stroke-width:2px;
classDef scheduler fill:#FFA07A,stroke:#FF4500,stroke-width:2px;
classDef queueManager fill:#9370DB,stroke:#6A5ACD,stroke-width:2px;
classDef queues fill:#DDA0DD,stroke:#BA55D3,stroke-width:1px;
classDef workerPool fill:#FFB6C1,stroke:#FF69B4,stroke-width:2px;
classDef redis fill:#FFC0CB,stroke:#DC143C,stroke-width:2px;
classDef eventBus fill:#ADD8E6,stroke:#4682B4,stroke-width:2px;
classDef wasm fill:#F0E68C,stroke:#FFD700,stroke-width:2px;
%% Nodes
Client[Client Applications]:::client --> API[API Gateway]:::api
API --> TaskManager[Task Manager]:::taskManager
subgraph Core System
TaskManager --> Scheduler[Task Scheduler]:::scheduler
TaskManager --> QueueManager[Queue Manager]:::queueManager
QueueManager --> Queue1[Priority Queue]:::queues
QueueManager --> Queue2[FIFO Queue]:::queues
QueueManager --> Queue3[LIFO Queue]:::queues
Queue1 --> WorkerPool[Worker Pool]:::workerPool
Queue2 --> WorkerPool
Queue3 --> WorkerPool
WorkerPool --> Worker1[Worker 1]
WorkerPool --> Worker2[Worker 2]
WorkerPool --> Worker3[Worker 3]
end
subgraph Storage Layer
Redis[(Redis Cluster)]:::redis
WorkerPool --> Redis
QueueManager --> Redis
TaskManager --> Redis
end
subgraph Event System
EventBus[Redis Pub/Sub]:::eventBus
WorkerPool --> EventBus
QueueManager --> EventBus
TaskManager --> EventBus
end
subgraph Language Bridge
Worker1 --> WASM[WASM Runtime]:::wasm
Worker2 --> WASM
Worker3 --> WASM
end
All the code-examples are just a bluff as of now
Perfect for distributed systems where tasks need to be processed across different services:
// Create a task chain
task := ottermq.NewTask().
Then("validate-order").
Then("process-payment").
Then("update-inventory").
Then("send-confirmation")Ideal for handling time-consuming tasks:
// Schedule a periodic task
scheduler.Every(24 * time.Hour).
At("02:00").
Run("generate-daily-reports")Great for systems requiring real-time event processing:
// Subscribe to task events
events.Subscribe("task.completed", func(e Event) {
log.Printf("Task %s completed: %v", e.TaskID, e.Result)
})Write workers in any language that compiles to WebAssembly:
// TypeScript worker
export function processTask(payload: any): Result {
// Process task
return { status: "success", data: processed }
}# Python worker (compiled to WASM)
def process_task(payload):
# Process task
return {"status": "success", "data": processed}OtterMQ can be configured via environment variables, config file, or CLI flags:
# config.yaml
redis:
host: localhost
port: 6379
password: ""
queues:
default:
type: fifo
max_size: 1000
worker_count: 5
priority:
type: priority
levels: 5
worker_count: 10
monitoring:
metrics_port: 9090
tracing_enabled: trueOtterMQ provides comprehensive monitoring capabilities:
- Prometheus metrics endpoint
- Grafana dashboards
- Distributed tracing with OpenTelemetry
- Real-time queue statistics
# Clone the repository
git clone https://github.com/fofsinx/ottermq.git
# Install dependencies
go mod download
# Run tests
make test
# Build
make buildContributions are welcome! Please read our Contributing Guide for details on our code of conduct and the process for submitting pull requests.
- Go 1.21 or higher
- Redis 6.x or higher
- Docker (for development environment)
OtterMQ is released under the MIT License. See the LICENSE file for details.