Open-source, high-performance LLM gateway written in Rust. Connect to any LLM provider with a single API. Observability Included.

• Multi-Provider Support: 143+ LLM providers via liter-llm integration
• OpenAI Compatible API: Drop-in replacement for OpenAI API calls
• Two Deployment Modes:
  • YAML Mode: Simple static configuration with config files
  • Database Mode: Dynamic configuration with PostgreSQL and Management API
• Built-in Observability: OpenTelemetry tracing and Prometheus metrics
• High Performance: Written in Rust with async/await support
• Hot Reload: Dynamic configuration updates (Database mode)
• Pipeline System: Extensible request/response processing
• Workspace Architecture: Modular design with hub-core, hub-gateway, hub-management

# Clone and build
git clone https://github.com/longcipher/aihub.git
cd aihub
cargo build --release

# YAML Mode
./target/release/hub

# Database Mode  
HUB_MODE=database DATABASE_URL=postgresql://user:pass@host:5432/db ./target/release/hub

The project uses a Cargo workspace architecture:

aihub/
├── bin/hub/                    # Binary crate
│   └── src/main.rs             # Application entry point
├── crates/
│   ├── hub-core/               # Core library
│   │   ├── config/             # Configuration management
│   │   ├── provider/           # LLM provider adapter
│   │   ├── state.rs            # Application state management
│   │   └── types/              # Shared type definitions
│   ├── hub-gateway/            # Gateway library
│   │   ├── routes.rs           # HTTP routing
│   │   └── pipeline/           # Request processing pipelines
│   └── hub-management/         # Management API library
│       ├── api/                # REST API endpoints
│       ├── db/                 # Database models and repositories
│       ├── services/           # Business logic
│       └── dto.rs              # Data transfer objects
├── migrations/                 # Database migrations
├── tests/                      # Integration tests
└── config.yaml                 # Example configuration

Perfect for simple deployments and development environments.

Features:

• Static configuration via config.yaml
• No external dependencies
• Simple provider and model setup
• No management API
• Single port (3000)

Example config.yaml:

providers:
  - key: openai
    type: openai
    api_key: sk-...

models:
  - key: gpt-4
    type: gpt-4
    provider: openai

pipelines:
  - name: chat
    type: Chat
    plugins:
      - ModelRouter:
          models: [gpt-4]

Ideal for production environments requiring dynamic configuration.

Features:

• PostgreSQL-backed configuration
• REST Management API (/api/v1/management/*)
• Hot reload without restarts
• Configuration polling and synchronization
• Dual ports (3000 for Gateway, 8080 for Management)

Setup:

1. Set up PostgreSQL database

2. Run migrations: sqlx migrate run

3. Set environment variables:

   HUB_MODE=database
   DATABASE_URL=postgresql://user:pass@host:5432/db

Port 3000:

• POST /api/v1/chat/completions - Chat completions
• POST /api/v1/completions - Text completions
• POST /api/v1/embeddings - Text embeddings
• GET /health - Health check
• GET /metrics - Prometheus metrics
• GET /swagger-ui - OpenAPI documentation

Port 8080:

• GET /health - Management API health check
• GET|POST|PUT|DELETE /api/v1/management/providers - Provider management
• GET|POST|PUT|DELETE /api/v1/management/model-definitions - Model management
• GET|POST|PUT|DELETE /api/v1/management/pipelines - Pipeline management

providers:
  - key: openai
    type: openai
    api_key: sk-...
    # Optional
    organization_id: org-...
    base_url: https://api.openai.com/v1

providers:
  - key: anthropic
    type: anthropic
    api_key: sk-ant-...

providers:
  - key: azure
    type: azure
    api_key: your-key
    resource_name: your-resource
    api_version: "2023-05-15"

providers:
  - key: bedrock
    type: bedrock
    region: us-east-1
    # Uses IAM roles or AWS credentials

Supports two authentication modes that route to different Google APIs:

# Option 1: API Key (uses Gemini Developer API)
providers:
  - key: vertexai
    type: vertexai
    api_key: your-gemini-api-key

# Option 2: Service Account (uses Vertex AI)
providers:
  - key: vertexai
    type: vertexai
    project_id: your-project
    location: us-central1
    credentials_path: /path/to/service-account.json

• Rust 1.88+
• PostgreSQL 12+ (for database mode)
• sqlx-cli (for migrations)

# Build
cargo build

# Test
cargo test

# Format
cargo fmt

# Lint
cargo clippy

# Run YAML mode
cargo run

# Run database mode
HUB_MODE=database DATABASE_URL=postgresql://... cargo run

# Install sqlx-cli
cargo install sqlx-cli --no-default-features --features postgres

# Run migrations
sqlx migrate run

Configure OTLP endpoint for distributed tracing:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

Available at /metrics:

• Request counts and latencies
• Provider-specific metrics
• Error rates
• Active connections

For metrics monitoring, use OpenObserve:

1. Deploy OpenObserve
2. Configure Prometheus remote write to OpenObserve
3. Query metrics in OpenObserve dashboard

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Client App    │───▶│     AiHub        │───▶│   LLM Provider  │
└─────────────────┘    │                  │    │  (OpenAI, etc.) │
                       │  ┌─────────────┐ │    └─────────────────┘
                       │  │ Config Mode │ │
                       │  │ YAML | DB   │ │    ┌─────────────────┐
                       │  └─────────────┘ │───▶│   Observability │
                       │                  │    │ (OTel, Metrics) │
                       │  ┌─────────────┐ │    └─────────────────┘
                       │  │ Management  │ │
                       │  │ API (DB)    │ │
                       │  └─────────────┘ │
                       └──────────────────┘

Licensed under the MIT License. See LICENSE for details.

We welcome contributions! Please see our Contributing Guide for details.