Track every job. Manage every lifecycle.

Better tracking and management of distributed job lifecycles — from web console.

---

scherry is a production-ready distributed job orchestration library for Go that lets you split large workloads into parallel tasks, distribute them over Kafka, and track every job and task throughout its lifecycle.

Unlike single-level task queues that leave you guessing when a batch run stalls or fails, scherry gives you a two-level job → task model on top of Asynq: split work once, execute tasks in parallel, persist state at every step, and inspect progress, retries, and failures from a single UI.

• How it builds on Asynq
• Feature comparison
• Architecture
• Features
• Install
• Prerequisites
• Configuration
• Quick start
  • Scheduled job (cron-triggered)
  • Manual job (on-demand trigger)
• Sharing a consumer across jobs
• Kafka tuning
  • ConsumerConfig
  • WriterConfig
  • TLS and SASL
• History console
  • Overview tab
  • History tab
  • Jobs tab
  • Upcoming tab
• Core interfaces
  • JobExecutor
  • TaskData
  • TaskExecutor
• Logger
• Hooks
• Payroll example
• Migrating an existing app
• Job lifecycle
• Task lifecycle
• API summary
• Contributing
• Third-Party Licenses
• License
• Credits

Asynq is a Redis-backed distributed task queue: you enqueue individual tasks, workers pick them up, and Asynq handles retries, priorities, and scheduling. It is excellent for simple background job processing.

scherry uses Asynq for one specific responsibility — cron-based job triggering — and then takes over for everything else:

The key additions over raw Asynq are:

• Two-level job → task model — A JobExecutor splits work into N tasks; a TaskExecutor handles one task. The library orchestrates splitting, persistence, fan-out, retries, and job finalization automatically.
• Kafka fan-out for tasks — Tasks are published to Kafka topics and consumed by scalable consumer groups, decoupling task throughput from the Redis-based job trigger layer.
• Durable Postgres persistence — Every job and task is written to partitioned scherry_jobs / scherry_tasks tables, giving you a reliable audit trail that survives Redis restarts.
• FSM-based lifecycle — Jobs and tasks move through an explicit finite-state machine (PENDING → RUNNING → COMPLETED / FAILED / etc.), with well-defined terminal states and automatic job finalization when all tasks settle.
• Retry topic + DLQ — Failed tasks are re-published to a dedicated retry topic (with configurable delay) rather than being held in Redis. Tasks that exhaust all retries are published to a DLQ topic for inspection.
• Manual / on-demand jobs — RegisterManual + Trigger lets you fire a job from an HTTP handler or event without a cron schedule.
• Task distribution keys — An optional Kafka partition key (DistributionKey) routes related tasks to the same partition/consumer, preserving ordering and enabling consumer affinity.
• Duplicate-run suppression — ExecuteJob reuses an existing run with the same caller-supplied refID, so a re-fired trigger does not start a second run. Best-effort rather than a guarantee: concurrent triggers sharing a refID can still both create a run (see Idempotency limitations).
• Execution hooks — OnTasksPublished, OnTaskStarted, OnTaskFinished, OnJobFinished callbacks for Prometheus metrics or any custom instrumentation.
• History console — A built-in web UI served on its own port, backed by Postgres, showing job/task progress, timings, retry lineage, and a one-click retry action.

The table below compares scherry against Asynq (which it wraps) and four other commonly used Go job/task queue libraries.

When to choose scherry: if your workload involves large batch jobs (payroll runs, data exports, bulk notifications) that need to be split into thousands of parallel tasks, tracked durably in Postgres, retried reliably via Kafka, and inspected through a history UI — with the cron trigger layer staying lightweight on Redis.

When Asynq alone is enough: for simple background task processing where a single-level queue over Redis is sufficient and you do not need durable Postgres history or Kafka-based fan-out.

When River fits better: if your entire stack already relies on Postgres and you want transactional job enqueueing without a separate broker; River Pro is required for workflow orchestration and guaranteed sequential execution.

When Machinery fits better: if you need broker flexibility (RabbitMQ, AWS SQS) or workflow primitives like chains and chords across a polyglot environment.

When go-quartz fits better: if you need a lightweight, zero-dependency cron/interval scheduler with no external infrastructure; go-quartz is ideal for simple recurring tasks within a single process where durable history, fan-out, retries, and a broker are not required.

Entry points

Retry and DLQ flow

When TaskExecutor.Execute returns an error, the library checks the task's remaining retry count:

1. Retries remaining — the task status is reset to PENDING and the message is re-published to the dedicated retry topic after the configured RetryDelay. A separate retry consumer group picks it up and calls ExecuteTask again.
2. No retries left — the task transitions to MAX_RETRIES_EXHAUSTED and the message is published to the DLQ topic for offline inspection or manual replay. The task is not re-processed automatically.

Job completion checks

A job is finalized (COMPLETED, PARTIAL_FAILED, or FAILED) once every one of its tasks has reached a terminal state (COMPLETED, MAX_RETRIES_EXHAUSTED, CANCELLED, REJECTED). Rather than re-evaluating this after every single task completion, JobConfig.JobCompletionCron — a required cron expression — periodically batch-checks that job's RUNNING instances and finalizes any that are ready, the same way Schedule triggers a job's execution on a cron. Register and RegisterManual both reject a JobConfig that leaves JobCompletionCron empty. You can also call Service.CheckJobCompletions(ctx, jobName) directly (e.g. from your own scheduler or an admin action) whenever you want a check to run outside the configured cron. A job created with zero tasks is always finalized immediately, since there would otherwise be nothing to ever trigger it.

Scheduling and triggering

• Cron-scheduled jobs — Register with a cron expression; Asynq fires the job automatically at the configured schedule
• Manual / on-demand jobs — RegisterManual + Trigger to fire a job from an HTTP handler, event stream, or any programmatic call
• CancelPrevious — when enabled, starting a new run of a job automatically cancels any previous in-flight run of the same job
• Enabled callback — a per-job func() bool evaluated at execution time; returning false rejects the run without removing the registration

Execution and fan-out

• Job → task splitting — JobExecutor.Execute returns a []TaskData; each item becomes one persisted task and one Kafka message
• Parallel task execution — tasks are consumed by a dedicated Kafka consumer group per job; task throughput scales independently of the Redis/Asynq layer
• Duplicate-run suppression — ExecuteJob reuses an existing run with the same caller-supplied refID; best-effort, so triggers that overlap in time can still both create a run (see Idempotency limitations)
• Task distribution keys — optional DistributionKey on each TaskData routes related tasks to the same Kafka partition, preserving ordering and enabling consumer affinity (e.g. all tasks for a given customer handled by one worker instance)
• Shared consumers — RegisterShared explicitly couples two or more jobs onto one Kafka consumer, one producer, and one TaskExecutor when you want to pool Kafka resources instead of running an isolated consumer per job (see Sharing a consumer across jobs)

Reliability and retries

• Retry topic with delay — on failure, tasks with retries remaining are re-published to a dedicated Kafka retry topic after the configured RetryDelay, then consumed and re-executed automatically
• Dead-letter queue (DLQ) — tasks that exhaust all retry attempts transition to MAX_RETRIES_EXHAUSTED and are published to a DLQ Kafka topic for offline inspection or manual replay
• Programmatic job retry — RetryJob clones a terminal job and re-queues its failed/exhausted tasks; fullRetry: true re-runs every task in the original job

Observability

• Durable Postgres history — every job and task is persisted to partitioned scherry_jobs / scherry_tasks tables, including execution timing (started_at, finished_at, duration_ms)
• FSM-based lifecycle — jobs and tasks move through an explicit finite-state machine with well-defined transitions; the library auto-finalizes a job once all its tasks reach a terminal state
• Execution hooks — OnTasksPublished, OnTaskStarted, OnTaskFinished, OnJobFinished callbacks for Prometheus metrics or any custom instrumentation
• History console — built-in web UI served on its own port, backed by Postgres; shows per-job progress, timings, retry lineage, and a one-click retry action

Extensibility

• Pluggable transport — inject custom Publisher and MessageReader implementations to replace the default kafka-go producer and consumer
• Kafka tuning — pass-through ConsumerConfig / WriterConfig for TLS, SASL, batching, compression, and all kafka-go producer/consumer knobs
• Programmatic task queries — GetJobTasks returns paginated, sequence-ordered tasks directly in Go without going through the HTTP console API

go get github.com/zepto-labs/scherry

• PostgreSQL with partitioned scherry_jobs and scherry_tasks tables
• Redis for Asynq scheduler/worker
• Kafka for task fan-out

Apply migrations from migrations/.

See docs/CONFIGURATION.md for the full bootstrap, job, and example environment variable reference.

svc, err := scherry.New(scherry.Config{
    ServiceName: "my-service",
    Logger:      myLogger,              // required
    Hooks:       myHooks,               // optional instrumentation
    Redis:       scherry.RedisConfig{Addr: "localhost:6379"},
    Database:    scherry.DatabaseConfig{Host: "localhost", Port: 5432, User: "postgres", Password: "secret", DBName: "jobs"},
})

err = svc.Register(scherry.JobConfig{
    Name:              "my_job",
    JobExecutor:       myJobExecutor,
    TaskExecutor:      myTaskExecutor,
    Schedule:          "0 */4 * * *",
    JobCompletionCron: "*/2 * * * *", // check for completed jobs every 2 minutes
    Retry:             scherry.RetryConfig{MaxRetries: 3, RetryDelay: 5 * time.Minute},
    AsynqOptions:      []asynq.Option{asynq.MaxRetry(3)},
    Kafka: scherry.KafkaConfig{
        Brokers:        []string{"localhost:9092"},
        TaskTopic:      "my_job_tasks",
        TaskRetryTopic: "my_job_tasks_retry",
        TaskDLQTopic:   "my_job_tasks_dlq",
        ConsumerGroup:  "my_job_worker",
        // ConsumerConfig and WriterConfig are optional — see "Kafka tuning"
        // below for examples.
    },
})

mux := asynq.NewServeMux()
svc.Start(ctx, mux)

server := asynq.NewServer(redisCfg.AsynqClientOpt(), asynq.Config{Concurrency: 5})
server.Run(mux)

Use RegisterManual for jobs that should only run when explicitly triggered (e.g. from an HTTP handler or event):

err = svc.RegisterManual(scherry.JobConfig{
    Name:              "my_manual_job",
    JobExecutor:       myJobExecutor,
    TaskExecutor:      myTaskExecutor,
    Retry:             scherry.RetryConfig{MaxRetries: 3, RetryDelay: 5 * time.Minute},
    JobCompletionCron: "*/2 * * * *", // check for completed jobs every 2 minutes
    Kafka: scherry.KafkaConfig{
        Brokers:        []string{"localhost:9092"},
        TaskTopic:      "my_manual_job_tasks",
        TaskRetryTopic: "my_manual_job_tasks_retry",
        TaskDLQTopic:   "my_manual_job_tasks_dlq",
        ConsumerGroup:  "my_manual_job_worker",
    },
})

// Fire on demand — e.g. from an HTTP handler:
refID := uuid.New().String()
err = svc.Trigger(ctx, "my_manual_job", refID, map[string]interface{}{"key": "value"})

By default every registered job gets its own Kafka consumer (one reader goroutine and consumer group per job), which gives each job full isolation. When you have several jobs whose task volume is low, or where you would rather pool Kafka resources than isolate them, RegisterShared lets two or more jobs share a single consumer, a single producer, and a single TaskExecutor.

A common motivation is limiting the load a job's consumer places on a downstream service. Each consumer is an independent source of concurrent calls to whatever your TaskExecutor talks to (a database, a rate-limited third-party API, an internal microservice). Running a separate consumer per job multiplies that pressure — N jobs mean N sets of in-flight requests hitting the same downstream. Coupling related jobs onto one shared consumer collapses them into a single, bounded stream of task processing against that dependency, so you can size and rate-limit the downstream for one consumer instead of many.

Coupling is explicit and only happens through RegisterShared — reusing the same ConsumerGroup/topic on separate Register/RegisterManual calls does not couple jobs and is rejected, so you can never share a consumer by accident.

err := svc.RegisterShared(scherry.SharedKafkaConfig{
    // One common Kafka config for every coupled job: brokers, topics,
    // consumer group, and consumer/writer tuning.
    Kafka: scherry.KafkaConfig{
        Brokers:        []string{"localhost:9092"},
        TaskTopic:      "billing_tasks",
        TaskRetryTopic: "billing_tasks_retry",
        TaskDLQTopic:   "billing_tasks_dlq",
        ConsumerGroup:  "billing_shared_worker",
    },
    // One shared TaskExecutor handles every task, regardless of which job
    // produced it (the consumer, and therefore its processing logic, is shared).
    TaskExecutor: billingTaskExecutor,
    // The coupled jobs. Each keeps its own trigger/split side; their Kafka and
    // TaskExecutor fields must be left empty (inherited from above).
    Jobs: []scherry.JobConfig{
        {
            Name:              "monthly_payroll_job",
            Schedule:          "0 */4 * * *",          // scheduled
            JobExecutor:       payrollExecutor,
            JobCompletionCron: "*/2 * * * *",
            Retry:             scherry.RetryConfig{MaxRetries: 3, RetryDelay: 5 * time.Minute},
        },
        {
            Name:              "bonus_payout_job",       // manual (no Schedule)
            JobExecutor:       bonusExecutor,
            JobCompletionCron: "*/2 * * * *",
            Retry:             scherry.RetryConfig{MaxRetries: 5, RetryDelay: time.Minute},
        },
    },
})

What is shared vs. per-job

Because task-to-job routing is resolved from Postgres (each consumed message is looked up by task ID), the single shared consumer dispatches every message to the correct job's lifecycle automatically — the jobs stay fully independent in history, scheduling, and retry policy.

Requirements and validation. RegisterShared requires at least two jobs, a non-empty Kafka.Brokers/TaskTopic/ConsumerGroup, a non-nil shared TaskExecutor, and a JobCompletionCron on every job. Each job's Kafka, TaskExecutor, Publisher, and reader fields must be left empty (they come from the shared config); setting any of them is a validation error.

Trade-offs. Sharing a consumer means the coupled jobs share a failure domain and throughput — you cannot scale one job's consumers independently, and because the retry consumer waits out a task's RetryDelay inline, a long delay on one job's task can hold up others on the same shared consumer. Prefer the default one-consumer-per-job when you need isolation; use RegisterShared when pooling Kafka consumer resources is the goal.

KafkaConfig exposes two optional pass-through fields that map directly to the underlying kafka-go types. The five routing fields (Brokers, TaskTopic, TaskRetryTopic, TaskDLQTopic, ConsumerGroup) are always managed by the library. Everything else is forwarded verbatim to kafka-go, which applies its own defaults for any zero value. Retry policy (MaxRetries, RetryDelay) is configured together via JobConfig.Retry (a RetryConfig) rather than on KafkaConfig — see the Quick start examples above.

Tunes both the main task consumer and the retry consumer. Brokers, GroupID, and Topic are set by the library and cannot be overridden here.

import kafkago "github.com/segmentio/kafka-go"

Kafka: scherry.KafkaConfig{
    // ... routing fields ...
    ConsumerConfig: kafkago.ReaderConfig{
        MinBytes:       1_000,                  // wait for ≥ 1 KB per fetch
        MaxBytes:       5_000_000,              // cap fetch response at 5 MB
        MaxWait:        500 * time.Millisecond, // max broker wait per fetch
        SessionTimeout: 30 * time.Second,       // rebalance window
        CommitInterval: time.Second,            // periodic auto-commit; 0 = sync per message
        StartOffset:    kafkago.LastOffset,     // skip historical messages for new groups
    },
},

Controls the producer. The library sets Addr from Brokers when Addr is nil. All other fields are used as-is. When WriterConfig is nil, a writer is created with a Hash balancer (for DistributionKey routing) and kafka-go defaults.

Kafka: scherry.KafkaConfig{
    // ... routing fields ...
    WriterConfig: &kafkago.Writer{
        Balancer:     &kafkago.Hash{},      // preserve DistributionKey routing
        RequiredAcks: kafkago.RequireAll,   // wait for all in-sync replica acks
        Compression:  kafkago.Zstd,        // compress batches
        BatchSize:    500,
        BatchTimeout: 5 * time.Millisecond,
    },
},

TLS and authentication are set via the native kafka-go fields inside ConsumerConfig and WriterConfig:

import (
    kafkago  "github.com/segmentio/kafka-go"
    "github.com/segmentio/kafka-go/sasl/plain"
)

Kafka: scherry.KafkaConfig{
    // ... routing fields ...
    ConsumerConfig: kafkago.ReaderConfig{
        Dialer: &kafkago.Dialer{
            TLS:           tlsCfg,
            SASLMechanism: plain.Mechanism{Username: "key", Password: "secret"},
        },
    },
    WriterConfig: &kafkago.Writer{
        Transport: &kafkago.Transport{
            TLS:  tlsCfg,
            SASL: plain.Mechanism{Username: "key", Password: "secret"},
        },
    },
},

The built-in history console is a dark-themed web UI served on its own port. It is backed entirely by the persisted scherry_jobs / scherry_tasks Postgres tables and auto-refreshes every 5 seconds. It has four tabs.

Start it alongside your application:

// Serve on its own port.
consoleServer, err := svc.StartConsole(":3002")
if err != nil {
    log.Fatal(err)
}
defer consoleServer.Shutdown(context.Background())

Or mount the handler under an existing mux (e.g. behind your own auth middleware):

mux := http.NewServeMux()
mux.Handle("/console/", http.StripPrefix("/console", svc.ConsoleHandler()))

Security: the console has no built-in authentication. Serve it on localhost during development, or protect it behind auth and network controls in production.

The landing tab. Shows aggregate metrics across all registered jobs for a selectable time range (last 1 h / 24 h / 7 d / 30 d).

• Job executions by status — stacked bar chart with one bar per job, broken down by execution status (COMPLETED, PARTIAL_FAILED, FAILED, RUNNING, PENDING, CANCELLED, REJECTED). Hover a segment for exact counts. Click legend entries to toggle individual statuses.
• Tasks over time — line chart of task outcomes (Succeeded / Failed / Other) bucketed by hour or day, filterable by job. Hover the chart for per-bucket counts.
• Jobs summary table — one row per registered job showing next scheduled run countdown, average job duration, total success and error counts, and error rate.

A paginated table of every job run, newest first. Filter by job name, execution status, reference ID (the caller-supplied refID used for deduplication — exact match), and page size. Because refID is not enforced as unique, this filter can return more than one run for a single reference ID.

Each row shows:

• Job name and a short job ID
• Status badge (color-coded)
• Progress bar segmented into completed (green) / failed (red) / running (amber) tasks, with a done/total label
• Start time and total duration

Click any row to open a slide-over detail drawer showing:

• Full job metadata: job ID, reference ID, parent job ID, start/finish timestamps, duration
• Expandable raw metadata JSON passed to the JobExecutor
• Retry controls — a "full retry" checkbox and a Retry job button (enabled only when the job is in a terminal state). Full retry re-runs every task; partial retry re-queues only MAX_RETRIES_EXHAUSTED tasks.
• Retry lineage — clickable links to every ancestor and descendant run in the retry chain
• Paginated task list with status filter, showing: sequence number, task name, status badge, attempt / max-retries, duration, and expandable result JSON

A static table of every job currently registered with the service — both scheduled and manual.

Columns: job name, cron schedule (blank for manual jobs), enabled status, max retries, cancel-previous flag, task topic, retry topic, DLQ topic, consumer group, and broker addresses.

Use this tab to verify that all jobs were registered with the expected Kafka and retry configuration.

Shows the next N scheduled fires for each cron-scheduled job (configurable: 3 / 5 / 10 / 20 runs per job), computed from the registered cron expressions.

Columns: time until the run (relative, e.g. "4m 12s"), absolute run-at timestamp, job name, and schedule expression.

Manual jobs (RegisterManual) have no schedule and do not appear here.

Splits a scheduled job into independent tasks:

type JobExecutor interface {
    Execute(ctx context.Context, metadata map[string]interface{}) ([]TaskData, error)
}

The unit returned by JobExecutor.Execute. Each element becomes one persisted task and one Kafka message:

type TaskData struct {
    Name            string
    Params          map[string]interface{}
    DistributionKey string
}

DistributionKey is an optional Kafka partition key. When set, all tasks that share the same key are routed to the same Kafka partition, preserving relative ordering and enabling consumer affinity (e.g. all tasks for a given account always land on the same worker). When left empty, no message key is set and the producer distributes tasks round-robin across partitions.

Typical uses:

• Customer/account isolation — set DistributionKey to a customer ID so that all tasks for that customer are processed sequentially by one consumer instance.
• Batch grouping — set it to a batch identifier when the order of tasks within a batch matters.
• Even distribution — leave it empty (default) when tasks are independent and you want maximum parallelism.

Runs a single task unit and returns an optional result payload that is persisted alongside the task record:

type TaskExecutor interface {
    Execute(ctx context.Context, params map[string]interface{}) (map[string]interface{}, error)
}

Implement scherry.Logger or use the built-in slog adapter:

logger := scherry.NewSlogLogger(slog.Default())

All four hooks are optional. Unset hooks are silently no-ops.

hooks := scherry.Hooks{
    OnTasksPublished: func(ctx context.Context, jobName string, count int) {
        tasksPublished.WithLabelValues(jobName).Add(float64(count))
    },
    OnTaskStarted: func(ctx context.Context, jobName, taskID string, attempt int) {
        tasksInFlight.WithLabelValues(jobName).Inc()
    },
    OnTaskFinished: func(ctx context.Context, jobName, taskID, status string, d time.Duration) {
        tasksInFlight.WithLabelValues(jobName).Dec()
        taskDuration.WithLabelValues(jobName, status).Observe(d.Seconds())
    },
    OnJobFinished: func(ctx context.Context, jobName, refID, status string, d time.Duration) {
        jobDuration.WithLabelValues(jobName, status).Observe(d.Seconds())
    },
}

See examples/payroll/ for two jobs running side-by-side:

• payroll JobExecutor loads employees and batches them into salary payout tasks
• payroll TaskExecutor simulates a bank transfer per batch
• bonus JobExecutor reads bonus_pct from metadata, batches employees
• bonus TaskExecutor computes and records a bonus transfer per employee
• main.go wires Kafka, Redis, Postgres, slog logger, hooks, and the history console

Run locally with Docker (applies migrations automatically):

cd examples/payroll
docker compose up -d --wait
go run .

Then open:

• http://localhost:3002 — history console (Postgres job/task history, progress, timings, retry)
• POST http://localhost:8080/bonus/trigger — fire an on-demand bonus payout

See examples/payroll/README.md for the full walkthrough.

1. Add scherry as a dependency
2. Implement JobExecutor / TaskExecutor with your existing business logic
3. Inject Kafka, Redis, Postgres, Logger, and Hooks at bootstrap
4. Call Register instead of your in-app scheduler registration
5. Run Asynq worker with the mux returned from Start

Your app keeps ownership of SQL, metrics, and domain code — the library owns orchestration only.

Jobs move through the FSM in internal/domain/state.go:

A job with all tasks in a terminal state is finalized on the next JobCompletionCron tick (or immediately, for a job created with zero tasks). See Job completion checks above. RetryJob requires the original job to be in a terminal state.

Tasks move through the FSM in internal/domain/state.go:

Typical happy path: PENDING → RUNNING → COMPLETED.

On failure with retries left: RUNNING → FAILED → PENDING (re-published to the retry topic with delay) → RUNNING → …

On failure with no retries left: RUNNING → FAILED → MAX_RETRIES_EXHAUSTED (published to DLQ).

Terminal task statuses (COMPLETED, FAILED, MAX_RETRIES_EXHAUSTED, REJECTED, CANCELLED) are used to determine when the parent job can be finalized.

Each task also carries a sequence_number — the 0-based index of the task in the []TaskData slice originally returned by JobExecutor.Execute. Use it (or the GetJobTasks method) to retrieve tasks in generation order. See Task ordering for why created_at is not reliable.

See the contributing guide to learn how to contribute to the repository and the development workflow.

This project depends on the following third-party packages. Their respective licenses apply to those packages.

• Agnish Upadhyay
• Shobhit Agarwal
• Kundini Kamat
• Shivam Singhal

• Koushik Kottamasu
• Nikhil Mittal