llm-go

llm-go is a provider-neutral Go library for model generation, streaming, tool use, multimodal messages, reasoning, and usage accounting. Provider SDK types stay behind a small common API, so applications can switch protocols without changing their conversation model.

Requires Go 1.27 or newer. The public API is still under active development.

• One request, response, tool, error, and model contract across providers
• Synchronous generation and lifecycle-safe streaming with typed events
• Multi-turn reasoning replay, prompt caching, and usage/cost accounting
• JSON Schema coercion and validation, plus tolerant JSON repair helpers
• Immutable model catalogs, provider profiles, and extensible protocol factories
• Concurrency-safe credentials, OAuth refresh, retries, and token budgeting
• Caller-owned HTTP clients, contexts, and returned data

go get github.com/XiaoConstantine/llm-go

llm-go separates wire-protocol adapters from model and provider metadata. Every adapter supports generation, typed stream events, classified errors, and token usage. In the matrix below, ✓ means the adapter implements the feature and — means it does not. Model-scoped features must still be declared by the configured model; a ✓ does not imply that every model exposed by an endpoint supports it.

JSON mode refers to llm.ResponseFormatJSON, not tool argument schemas. Tool calls work in generation and streaming, including partial argument events where the protocol exposes them. Strict tools and reasoning replay are model- and protocol-dependent. Cache retention markers are emitted only where supported; unsupported cache keys and session-affinity mappings fail before provider I/O. OpenAI Responses also supports durable background jobs through llm.BackgroundGenerator.

openai/codex.Config.Transport supports these modes; the default remains SSE.

Cached sockets require Request.SessionID, are isolated by client, session, account, and credential. WebSocketMaxSessions defaults to 64; busy overflow and requests with per-attempt headers use transient sockets. Use WebSocketDialer for WebSocket-specific proxy, mTLS, or custom dialing. Client.Close is idempotent and closes all owned sockets and timers.

These helpers are opt-in and do not change provider defaults.

Tool inputs use JSON Schema 2020-12 by default, support declared drafts and local references offline, and use Go/RE2-compatible patterns.

Generate, Stream, background responses, and CollectStructural(stream) return complete, structurally valid tool calls with their original arguments. They do not enforce argument schemas. Execution owners must call ValidateToolCalls(tools, calls) before invoking tools and can return an error ToolResult on rejection so the model can correct its call. Tool declarations, strict JSON, provider protocol checks, and constrained-generation settings are unchanged. ValidateToolCallStream remains an explicit schema-validation wrapper for callers that deliberately want schema failures to terminate collection.

Migration: automatic response argument-schema validation has been removed. Collect(stream, tools) retains its published signature and schema-validation behavior, including rejection of undeclared calls when tools is nil. Agent loops that need corrective tool results can use CollectStructural(stream) and validate arguments explicitly at the execution boundary. No request flag or diagnostic metadata is needed; do not execute an unchecked call just because generation succeeded.

models.NewFactoryRegistry creates an immutable registry containing the nine built-in protocol factories plus caller registrations. ProviderConfig.API remains the default route when ModelInfo.API is empty; AdditionalAPIs explicitly enables mixed-protocol models for the same provider. Factories receive owned construction snapshots and an optional live credential resolver.

models.CatalogManager keeps Catalog immutable while atomically publishing a static baseline plus validated provider overlays. Refresh restores persisted state before conditional fetches, supports ETag/Last-Modified, force and provider-selective refresh, and NoNetwork restore-only operation. Available uses CredentialManager and optional secret-free provider filters. Refreshes run only when called; the manager owns no permanent goroutines and needs no Close. The modelsdev package provides a CatalogSource implementation to dynamically fetch models from models.dev.

models.BuiltinProvider supplies protocol and endpoint defaults for these services:

Local servers that expose an OpenAI-compatible /v1/chat/completions endpoint use the openai adapter. They do not need a built-in profile, and the adapter permits an empty API key; authenticated servers can instead use an API key or custom headers.

Choose a provider ID, configure models.OpenAIChatCompletions, and supply the local model through llm.ModelInfo. Streaming, tools, JSON mode, vision, and reasoning depend on what that server and model implement, so declare only the capabilities they actually support. The same configuration works for other compatible servers.

The built-in catalog covers OpenAI, Anthropic, Amazon Bedrock, Gemini, Vertex AI, Mistral, Codex, and the OpenAI-compatible profiled providers. Azure Responses uses caller-supplied model metadata. Custom BaseURL, headers, and HTTP clients support compatible gateways.

The generated catalog can configure the protocol, capabilities, compatibility, and pricing for a model:

package main

import (
    "context"
    "log"
    "os"
    "time"

    llm "github.com/XiaoConstantine/llm-go"
    "github.com/XiaoConstantine/llm-go/models"
)

func main() {
    model, ok := models.BuiltinCatalog().Model("openai", "gpt-5-mini")
    if !ok {
        log.Fatal("model not found")
    }

    providers, err := models.New(models.ProviderConfig{
        ID:     "openai",
        API:    models.OpenAIResponses,
        APIKey: os.Getenv("OPENAI_API_KEY"),
    })
    if err != nil {
        log.Fatal(err)
    }

    generator, err := providers.GeneratorFor(model)
    if err != nil {
        log.Fatal(err)
    }

    ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
    defer cancel()

    response, err := generator.Generate(ctx, llm.Request{
        Messages: []llm.Message{{
            Role:    llm.RoleUser,
            Content: []llm.Part{{Text: "Say hello in one sentence."}},
        }},
    })
    if err != nil {
        log.Fatal(err)
    }
    log.Print(response.Text())
}

Concrete provider packages can also be constructed directly when protocol-level configuration is needed.

Every successful stream must be closed, including after Recv returns an error. Close is idempotent, unblocks a pending Recv, and waits for transport cleanup. A cleanup error returned by Close does not replace the stream's sticky terminal state.

stream, err := generator.Stream(ctx, request)
if err != nil {
    return err
}

for {
    chunk, recvErr := stream.Recv()
    if errors.Is(recvErr, io.EOF) {
        break
    }
    if recvErr != nil {
        _ = stream.Close()
        return recvErr
    }
    for _, part := range chunk.Content {
        fmt.Print(part.Text)
    }
}
if err := stream.Close(); err != nil {
    return err
}

Chunk.Events exposes typed lifecycle events for text, reasoning, and partial or completed tool calls. Completed values remain available through the ordinary chunk fields for response assembly.

Provider and protocol failures are classified as *llm.Error. Context cancellation, deadlines, custom cancellation causes, and underlying transport errors remain discoverable with errors.Is or errors.As. llm.IsContextOverflow handles both classified failures and providers that silently fill or truncate the context window.

The library does not impose a default request timeout. Supply an appropriate context deadline or configure the provided http.Client.

models.BuiltinCatalog() returns an immutable generated catalog whose read methods return independently owned model metadata. models.Catalog and models.Collection are immutable and safe for concurrent use when caller-supplied HTTP clients and credential resolvers are also safe.

models.CredentialStore supports API-key and OAuth credentials. models.MemoryCredentialStore provides an in-memory implementation, while models.CredentialManager coalesces concurrent refreshes, supports live credential rotation for long-lived generators, and safely handles token rotation. models.AnthropicOAuth and models.CodexOAuth expose non-interactive PKCE exchange/refresh primitives. Their subscription endpoints and client IDs are provider-private, unofficial, unstable, and replaceable; applications remain responsible for browser, callback, prompt UI, and persistent storage.

models.FindEnvAPIKeys lists configured variables in precedence order, models.EnvAPIKey returns the preferred value, and models.HasAmbientCredentials reports selected Vertex ADC and Bedrock environment or file hints without exposing secrets.

make fmt       # Format Go sources.
make fmt-check # Check formatting without changing files.
make lint      # Run the pinned golangci-lint version used by CI.
go test ./...
go test -race ./...
go vet ./...

The built-in catalog source is models/catalogsource/catalog.json. Regenerate and verify it with:

go generate ./models
git diff --exit-code -- models/catalog_generated.go

The Update model catalog GitHub Actions workflow checks models.dev every six hours and opens a tested pull request for Anthropic, direct OpenAI, and Google catalog updates. New entries must advertise tool support and text-only output; other model types and Codex subscription entries need separate maintenance. See the catalog maintenance guide for setup, manual runs, and reproducing an update from its saved snapshot.