This directory contains the nginx HTTP policy worker. In the shipped Compose topology, proxy-router/ owns the public ports and distributes TCP connections across multiple private instances of this image. The workers retain TLS, HTTP/2, CORS, rate limits, rewrites, and service routing.

Each nginx policy worker routes requests to backend services by URL path:

• /api/v1/ → Main application API v1 (proxies to backend /v1/)
• /v1/ → Direct API v1 (without /api/ prefix)
• /chain-rpc/ → Blockchain RPC endpoint (port 26657)
• /chain-api/ → Blockchain REST API (port 1317)
• /chain-grpc/ → Blockchain gRPC endpoint (port 9090)
• /jaeger/ → Jaeger UI when JAEGER_ENABLED=true and the observability overlay is running (nginx basic auth required)
• /grafana/ → Grafana UI when GRAFANA_ENABLED=true and the observability overlay is running (Grafana login required)
• /health → Policy readiness: nginx data listener and application-network DNS
• / → Explorer dashboard when DASHBOARD_PORT is set, otherwise a simple "dashboard not configured" page

1. Single Entry Point: Only one port (80) needs to be exposed externally
2. Simplified Networking: No need to manage multiple port mappings
3. Security: Internal services are not directly accessible from outside
4. Load Balancing: Can easily add multiple backend instances
5. SSL Termination: Easy to add HTTPS support in one place
6. Monitoring: Centralized access logs and metrics
7. Production Ready: Standard architecture pattern for containerized apps

• nginx.unified.conf.template - Unified nginx configuration template rendered via env vars
• entrypoint.sh - Script that substitutes environment variables and starts nginx
• setup-ssl.sh - Helper to fetch TLS certs from proxy-ssl when HTTPS is enabled
  • Modes: issue (default), renew, renew-if-needed (uses stored order.id; see Renewal)
• Dockerfile - Container image definition for the proxy service
• README.md - This documentation file

Key runtime environment variables:

Versiond-side (on the versiond container, not the join proxy):

Public observability paths that still include a version segment are rewritten internally to versionless canonical URIs (no client-visible redirect). Clients keep calling the old URLs; nginx drops the version segment before versiond so scrapers need not follow redirects and cannot bind protocol version via the path.

Prefer these URLs in new monitors / runbooks:

GET /devshard/sessions/{escrow_id}/diffs
GET /devshard/sessions/{escrow_id}/mempool
GET /devshard/sessions/{escrow_id}/signatures
GET /devshard/stats/shards
GET /devshard/stats/shards/{escrow_id}
GET /devshard/metrics
GET /devshard/healthz                 # versiond supervisor (not a child)
GET /devshard/{version}/healthz       # that child's healthz
GET /devshard/{version}/clock          # that child's clock (host-ping RTT/clock); not rewritten

Protocol traffic stays versioned: POST …/chat/completions, gossip, challenge-receipt, and GET …/payloads are not rewritten.

Public obs paths (versionless and rewritten legacy) use a dedicated nginx zone (devshard_obs, default 10r/s burst 20) so scrapers cannot amplify polling under the exempt chat limits. Chat / gossip / payloads remain on the exempt catch-all.

versiond serves the versionless obs paths:

• Session-scoped (/sessions/{id}/diffs|mempool|signatures, /stats/shards/{id}): when Postgres is configured (PGHOST / DATABASE_URL), route by sessions.version; unbound → 404. If lookup is disabled or PG errors, fan-out across children. Lookup errors emit a rate-limited warn (session version lookup failed; falling back to fan-out) and increment an in-process counter (proxy.LookupFanoutErrors) — they are not silent.
• Process-level (/metrics, /stats/shards list): pin to newest running version by numeric/dotted comparison (v10 > v2, v0.2.11 > v0.2.9), not lexicographic order.
• Health: GET /healthz on versiond is supervisor status (mux, ahead of the proxy). Join proxy GET /devshard/healthz hits that. Per-child health is GET /devshard/{version}/healthz (not rewritten). Do not use versionless /healthz to probe a specific child.

Disable lookup: VERSIOND_DISABLE_SESSION_LOOKUP=true.

Grafana dashboards in deploy/join/observability/ scrape Prometheus metrics from devshardd /metrics via service discovery — they do not call HTTP diffs URLs. For ad-hoc HTTP debugging of a shard, use the versionless paths above.

In the shipped HA topology this worker has one upstream, proxy:18081. The public proxy-router selects any route-ready inner router; every versiond-router independently applies the same multi-host stickiness and legacy SQLite pinning:

HA requests get Devshard-Ha: true; devshardd then requires DEVSHARD_STORAGE_MODE=postgres + PGHOST. See versiond-router/README.md.

/v1/ is split across two backends. The proxy registers exact/regex locations for read-only query paths before the generic /v1/ catch-all:

• edge-api (public Tier A) — status, models, pricing, participants (GET), epochs, restrictions, BLS, bridge addresses, poc-batches
• edge-api (optional) — verify-proof / verify-block / debug/*; private by default (EDGE_API_EXPOSE_OPTIONAL_ROUTES=false → 403). Opt in when needed; auth can be enforced in nginx before this proxy.
• dapi (api) — inference and node operations: chat/completions, inference payloads, PoC proofs, stats, bridge queue, participant registration (POST /v1/participants)
• versiond — devshard sessions: /v1/devshard/* is rewritten internally to /devshard/v1/*, then proxied like other /devshard/ traffic

/v1/participants is method-split: GET/HEAD/OPTIONS → edge-api; other methods (notably POST registration) → dapi via an internal named location. Without that split, nginx would send POST to edge-api and return 405.

To publish verify/debug on the public proxy:

export EDGE_API_EXPOSE_OPTIONAL_ROUTES=true
# Optional: front with nginx basic auth / allowlist before this container.

Multi-instance edge-api (deploy/join/docker-compose.edge-api-multi.yml):

Client -> proxy-router -> proxy-policy -> edge-api-router nginx -> edge-api-N

This release deliberately preserves the existing edge-api nginx router and does not require a new edge-api readiness contract. Moving that pool into the public HAProxy is a separate edge-api lifecycle change.

Jaeger and Grafana UIs are disabled by default (JAEGER_ENABLED=false, GRAFANA_ENABLED=false in deploy/join/config.env.template). The observability stack (Prometheus, Loki, trace export) can run without exposing UIs on the public proxy.

When enabling public UI routes, set credentials before flipping the enable flags:

1. Jaeger — Jaeger has no application login. Set JAEGER_BASIC_AUTH_USER and JAEGER_BASIC_AUTH_PASSWORD, then set JAEGER_ENABLED=true. The proxy refuses to start if Jaeger is enabled without basic auth credentials.
2. Grafana — Set a strong GRAFANA_ADMIN_PASSWORD (and optionally GRAFANA_ADMIN_USER), then set GRAFANA_ENABLED=true. The proxy refuses to start if Grafana is enabled with a missing or placeholder password (admin1, <FILLIN>, etc.).

Example (deploy/join/config.env):

export JAEGER_BASIC_AUTH_USER=jaeger
export JAEGER_BASIC_AUTH_PASSWORD='your-jaeger-basic-auth-secret'
export GRAFANA_ADMIN_USER=admin
export GRAFANA_ADMIN_PASSWORD='your-grafana-admin-secret'
export JAEGER_ENABLED=true
export GRAFANA_ENABLED=true

See docs/observability/observability-overview.md for the full join-stack setup.

Note: GLOBAL_RATE_LIMIT_RPS acts as a total ceiling for a single IP. It must be higher than your highest specific limit (e.g. higher than Exempt limit).

The two-tier join deployment terminates public connections at proxy-router. For that topology, configure the external load balancer to send PROXY protocol and set PROXY_ROUTER_PROXY_PROTOCOL_FROM; the private nginx policy workers trust only the PROXY header generated by that public tier. The PROXY_REAL_IP_FROM settings below apply when this nginx image itself is the public listener.

• PROXY_REAL_IP_FROM should include only the immediate proxy/LB hop(s) that connect to nginx.
• Prefer exact addresses or narrow CIDRs (/32) over broad private ranges.
• If your proxy is directly internet-facing (no upstream LB/reverse-proxy), leave PROXY_REAL_IP_FROM unset.

Example (Docker bridge gateway):

PROXY_REAL_IP_FROM=172.18.0.1/32
PROXY_REAL_IP_HEADER=X-Forwarded-For
PROXY_REAL_IP_RECURSIVE=off

Direct Docker proxy on a server (no ingress/LB in front):

# Best: disable real_ip parsing (nginx already sees client IP)
PROXY_REAL_IP_FROM=
PROXY_REAL_IP_HEADER=X-Forwarded-For
PROXY_REAL_IP_RECURSIVE=off

Docker proxy behind one trusted ingress/LB hop:

# Trust only the ingress/LB source IP(s)
PROXY_REAL_IP_FROM=172.18.0.1/32
PROXY_REAL_IP_HEADER=X-Forwarded-For
PROXY_REAL_IP_RECURSIVE=off

Docker proxy behind multiple trusted proxy layers:

# Only if every proxy in chain is trusted and controlled by you
PROXY_REAL_IP_FROM=10.42.16.0/20 10.42.32.0/20
PROXY_REAL_IP_HEADER=X-Forwarded-For
PROXY_REAL_IP_RECURSIVE=on

Avoid:

• Broad defaults like 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16.
• Enabling PROXY_REAL_IP_RECURSIVE=on unless you intentionally trust a full proxy chain.

• NGINX_MODE=http: listen on 80 only; SSL disabled.
• NGINX_MODE=https: listen on 443 with SSL; requires CERT_ISSUER_DOMAIN and a reachable proxy-ssl service to obtain certs if missing.
• NGINX_MODE=both: listen on 80 and 443; same SSL requirements as https.

When SSL is enabled, entrypoint.sh validates the local certificate/key pair before rendering nginx configuration. A valid pair continues without contacting proxy-ssl. A missing, truncated, or mismatched pair is repaired through setup-ssl.sh before the first nginx -t, including in HTTPS-only mode. Policy workers sharing one certificate volume serialize every setup-ssl.sh call through a shared flock, and the in-lock revalidation ensures only one worker places the initial order while its siblings adopt the published bundle; a worker whose pair is already valid starts without waiting for the lock.

If the certificate marker is missing while private.key and order.id remain, startup first recovers the certificate from that order. A new order is created only when the saved order cannot recover the local key.

If issuance fails at startup (for example, proxy-ssl is not reachable yet) and NGINX_MODE=both, the entrypoint temporarily serves HTTP only. A background worker keeps retrying with exponential backoff (starting at PROXY_SSL_RETRY_SECONDS, capped at the renewal interval) and, once a certificate is issued, re-renders the HTTPS configuration, validates it with nginx -t, and reloads nginx — port 443 comes back without a container restart. The same worker renews certificates issued through proxy-ssl (identified by a stored order.id) within RENEW_BEFORE_DAYS of expiry.

Valid manually supplied certificate/key pairs without order.id remain under operator management and are not sent to proxy-ssl. An invalid manual pair cannot serve HTTPS, so startup treats it as an incomplete bundle, obtains a replacement from proxy-ssl, and stores the resulting order.id for automatic renewal.

The worker also repairs legacy incomplete bundles. If HTTPS validation finds a truncated certificate or a certificate that does not match its private key, setup-ssl.sh repair first requests the certificate for the stored order.id and verifies that it belongs to the local key. It creates a new order only when the stored order cannot recover that key. After a valid pair is published, nginx is validated and reloaded automatically.

PROXY_SSL_RETRY_SECONDS and RENEW_INTERVAL_HOURS must be positive integers. Invalid values stop the container during configuration validation instead of starting a busy retry loop.

Below are minimal environment configurations for the compose stack under deploy/join/config.env. This section lists only environment variables; Docker commands are provided separately below.

NGINX_MODE=http
API_PORT=8000

NGINX_MODE=https
API_SSL_PORT=8443
CERT_ISSUER_DOMAIN=your.domain
CERT_ISSUER_JWT_SECRET=change-me
ACME_ACCOUNT_EMAIL=you@example.com
ACME_DNS_PROVIDER=<route53|cloudflare|gcloud|azure|digitalocean|hetzner>
# Provider credentials per your DNS (see proxy-ssl README)

Notes:

• The compose maps ports 80 and 443; with NGINX_MODE=https, nginx listens on 443 only.
• Certificates are stored under ./secrets/nginx-ssl (bind-mounted to /etc/nginx/ssl) and used automatically by proxy.

NGINX_MODE=both
API_PORT=8000
API_SSL_PORT=8443
CERT_ISSUER_DOMAIN=your.domain
CERT_ISSUER_JWT_SECRET=change-me
ACME_ACCOUNT_EMAIL=you@example.com
ACME_DNS_PROVIDER=<route53|cloudflare|gcloud|azure|digitalocean|hetzner>
# Provider credentials per your DNS (see proxy-ssl README)

Environment in deploy/join/config.env:

NGINX_MODE=https
API_SSL_PORT=8443
SERVER_NAME=your.domain
SSL_CERT_SOURCE=./secrets/nginx-ssl
# do not set CERT_ISSUER_DOMAIN when using manual certs

NGINX_MODE=both
API_PORT=8000
API_SSL_PORT=8443
SERVER_NAME=your.domain
SSL_CERT_SOURCE=./secrets/nginx-ssl
# do not set CERT_ISSUER_DOMAIN when using manual certs

This works with any DNS provider using an interactive DNS-01 challenge. Certbot will pause and show a TXT record to add at _acme-challenge.your.domain. Create that record in your DNS, wait for propagation, then press Enter to continue.

Recommended (one-shot, writes directly into the mounted directory deploy/join/secrets/nginx-ssl):

• Host-installed Certbot:

DOMAIN=your.domain
ACCOUNT_EMAIL=your_email@example.com
mkdir -p secrets/nginx-ssl secrets/certbot/{config,work,logs}
sudo certbot certonly --manual --preferred-challenges dns \
  --config-dir ./secrets/certbot/config \
  --work-dir ./secrets/certbot/work \
  --logs-dir ./secrets/certbot/logs \
  -d "$DOMAIN" \
  --email "$ACCOUNT_EMAIL" --agree-tos --no-eff-email \
  --deploy-hook 'install -m 0644 "$RENEWED_LINEAGE/fullchain.pem" ./secrets/nginx-ssl/cert.pem; install -m 0600 "$RENEWED_LINEAGE/privkey.pem" ./secrets/nginx-ssl/private.key'

• Dockerized Certbot (no host install needed):

DOMAIN=your.domain
ACCOUNT_EMAIL=your_email@example.com
mkdir -p secrets/nginx-ssl secrets/certbot
docker run --rm -it \
  -v "$(pwd)/secrets/certbot:/etc/letsencrypt" \
  -v "$(pwd)/secrets/nginx-ssl:/mnt/nginx-ssl" \
  certbot/certbot certonly --manual --preferred-challenges dns \
  -d "$DOMAIN" --email "$ACCOUNT_EMAIL" --agree-tos --no-eff-email \
  --deploy-hook 'install -m 0644 "$RENEWED_LINEAGE/fullchain.pem" /mnt/nginx-ssl/cert.pem; install -m 0600 "$RENEWED_LINEAGE/privkey.pem" /mnt/nginx-ssl/private.key'

Renewal: rerun the same one-shot command before expiry (manual DNS step required each time), then reload nginx (see Docker commands below).

Run from deploy/join after setting config.env.

• Prepare bind-mount directories (safe to rerun):

mkdir -p secrets/nginx-ssl secrets/certbot

• Initial start (enable HTTPS with proxy-ssl profile):

source ./config.env && \
docker compose --profile "ssl" -f docker-compose.yml -f docker-compose.mlnode.yml up -d

• Initial start with observability overlay:

source ./config.env && \
docker compose -f docker-compose.yml -f docker-compose.mlnode.yml -f docker-compose.observability.yml up -d

• Access the observability UIs through the proxy after startup:

${PUBLIC_URL}/jaeger/
${PUBLIC_URL}/grafana/

proxy, proxy-policy, and proxy-policy2 form one deployment unit. Host release updates apply the complete active Compose model: the base files plus the host's ML, versiond, edge-api, observability, managed-service, and operator overrides in their original order.

The release updater rolls this unit reserve-first. It replaces one policy worker, waits until the public HAProxy admits that exact container, replaces the other worker, and changes the public proxy generation last. proxy-router/test-routing.sh exercises continuity for non-idempotent requests while the replicated policy workers are replaced. The public proxy is a singleton: replacing it is a maintenance boundary and may interrupt active connections.

Rollback restores the captured image, environment, networks, capabilities, and Compose definition for the same deployment unit. The updater targets these services explicitly, so unrelated node services and overlay-owned containers remain in their existing model. Use the release's host updater and rollback instructions with the same config.env, Compose files, and profiles that describe the running host.

General operational guidance aligns with the Gonka Host Quickstart.

/health is served through nginx's production listener and returns 200 while the local policy sidecar can resolve the configured application-network alias. It returns 503 while that network is unavailable or the sidecar is restarting, allowing the public HAProxy to withdraw this policy worker.

1. Ensure NGINX_MODE is https or both and CERT_ISSUER_DOMAIN is set.
2. Verify proxy-ssl is running and reachable from proxy-policy (see proxy-ssl/README.md).
3. Check logs of proxy-policy for "SSL setup failed" or config validation errors.
4. Confirm DNS for SERVER_NAME/CERT_ISSUER_DOMAIN points to your proxy.

1. Check if the backend service is running: docker compose ps
2. Verify service names match the upstream definitions in nginx.conf
3. Check nginx logs: docker compose logs proxy-policy

WebSocket support is configured for RPC connections and dashboard hot-reloading. If you have issues:

1. Verify the Upgrade and Connection headers are properly set
2. Check if the backend service supports WebSockets

1. Adjust worker_connections in nginx.conf
2. Enable additional caching if needed
3. Monitor nginx access logs for slow requests

• X-Frame-Options: DENY
• X-Content-Type-Options: nosniff
• X-XSS-Protection: enabled
• Client body size limit: 10MB
• gzip compression enabled for better performance

If you're upgrading from a previous version with hardcoded ports:

1. Replace nginx.conf with nginx.unified.conf.template
2. Update your Dockerfile to use the new entrypoint
3. Add environment variables to your docker-compose.yml
4. Rebuild your nginx container

The entrypoint script provides sensible defaults, so existing setups will continue to work without changes.