Rust library providing a framework for building efficient web crawlers and asynchronous task processing systems with multiple backend support.
- Task Queues: Support for Fjall (default persistent backend) and in-memory storage
- Mailbox Runtime: Tower-native dispatcher/workers with retries, backpressure, and control signals
- Dead Letter Queue (DLQ): Automatic handling of failed tasks
- Round-Robin Processing: Fair task distribution across different domains
- Health Checks: Built-in monitoring capabilities
- Flexible Architecture: Modular design supporting custom task types and processors
Add to your Cargo.toml:
[dependencies]
capp = "0.6"
# Optional features
capp = { version = "0.6", features = ["router"] }use std::{sync::Arc, time::Duration};
use capp::{
manager::{MailboxConfig, ServiceRequest, build_service_stack, spawn_mailbox_runtime},
queue::{InMemoryTaskQueue, JsonSerializer, Task},
};
use serde::{Deserialize, Serialize};
use tower::{BoxError, service_fn};
#[derive(Debug, Clone, Serialize, Deserialize)]
struct TaskData {
value: u32,
}
#[tokio::main]
async fn main() -> Result<(), BoxError> {
let queue = Arc::new(InMemoryTaskQueue::<TaskData, JsonSerializer>::new());
let ctx = Arc::new(());
let service = build_service_stack(
service_fn(|req: ServiceRequest<TaskData, ()>| async move {
println!("processing {}", req.task.payload.value);
Ok::<(), BoxError>(())
}),
Default::default(),
);
let runtime = spawn_mailbox_runtime(
queue,
ctx,
service,
MailboxConfig {
worker_count: 4,
dequeue_backoff: Duration::from_millis(25),
..Default::default()
},
);
runtime
.producer
.enqueue(Task::new(TaskData { value: 42 }))
.await?;
runtime.shutdown().await;
Ok(())
}Configure via TOML files:
[app]
threads = 4
max_queue = 500
[http]
timeout = 30
connect_timeout = 10
[http.proxy]
use = true
uri = "http://proxy.example.com:8080"- http: HTTP client functionality with proxy support
- router: URL classification and routing
- healthcheck: System health monitoring
- cache: Cache helpers
- urls: URL parsing helpers
- stats-http: HTTP stats endpoint for mailbox runtime
- observability: OTLP metrics export
MIT
Contributions welcome! Please read non-existent guidelines and submit PRs.