One-shot Go job that syncs European Central Bank reference rates into PostgreSQL. Designed to run from an external scheduler (k8s CronJob, ECS task, cron) — the binary runs to completion and exits.
cp .env.template .env # edit DB_* values
make local-db-up # start a local Postgres in Docker
make local-run # full backfill against the local DBFor a quick daily refresh instead of a full backfill, set SYNC_MODE=daily_sync in .env (or use make local-run-daily).
Each invocation:
- Applies pending database migrations (skippable via
RUN_MIGRATIONS=false). - Seeds the
providerstable with ECB metadata; preserves externalCURAPIrows; removes any stale provider keys (logged at WARN). - Backfills missing ECB rates into
rates, resuming fromMAX(date)per provider. - Updates
currenciesandcurrency_coveragesfor any newly observed ISO codes. - Records the run in
sync_runsand emits a structuredmetriclog event. - Exits with
0on success,1on failure,130on signal-initiated shutdown.
Inserts use ON CONFLICT DO NOTHING, so re-running over already-stored dates is safe — there is no duplicate risk.
SYNC_MODE |
Use case | Default timeout | Lookback cap |
|---|---|---|---|
full |
Initial seeding or recovery — walks the whole history from providers.coverage_start. |
6h | none |
daily_sync |
Recurring ECB refresh (production cron). | 30m | DAILY_SYNC_LOOKBACK_DAYS (default 7) |
In daily_sync, if last_synced is older than the lookback cap, the run still only fetches the last N days. This keeps recurring jobs bounded after an outage.
Copy .env.template to .env. Every variable below is documented inline in the template too.
Loading order: process environment wins, then .env.local (developer overrides, gitignored), then .env (team baseline).
| Variable | Notes |
|---|---|
DB_USER, DB_PASSWORD, DB_NAME, DB_HOST, DB_PORT |
Standard Postgres connection. |
DB_SSLMODE |
Defaults to require (encrypts, but does not verify the server certificate). Use verify-full against an untrusted network — it authenticates the server and prevents MITM (provide the CA via sslrootcert in DATABASE_URL). Set to disable only for local Postgres without TLS. |
DATABASE_URL (alt.) |
Single DSN alternative. Takes precedence over the split DB_* vars when set. |
| Variable | Default | Notes |
|---|---|---|
SYNC_MODE |
daily_sync |
full or daily_sync |
DAILY_SYNC_TIMEOUT |
mode-dependent | Override only when needed. |
DAILY_SYNC_LOOKBACK_DAYS |
7 |
Cap on how far back a daily_sync run reaches. 0 disables. |
RUN_MIGRATIONS |
true |
Set false in production when migrations are run out-of-band. |
| Variable | Default |
|---|---|
DB_MAX_CONNECTIONS |
10 |
DB_MIN_CONNECTIONS |
5 |
DB_MAX_CONN_LIFETIME |
30m |
DB_MAX_CONN_IDLE_TIME |
5m |
DB_HEALTH_CHECK_PERIOD |
30s |
| Variable | Default | Notes |
|---|---|---|
LOG_LEVEL |
info |
debug, info, warn, error |
DEBUG |
true |
Gates scheduler heartbeat logs. Does not affect log level. |
DEBUG_HEARTBEAT_INTERVAL |
20s |
Interval for heartbeat logs when DEBUG=true. |
| Target | What it does |
|---|---|
make build |
Build all packages with version stamping. |
make release-build VERSION=v0.1.0 |
Build a stripped static linux binary into dist/. |
make test |
Run unit tests (no DB required). |
make test-integration |
Start local Postgres and run all tests including integration. |
make coverage |
Show per-package coverage. |
make sqlc-generate |
Regenerate typed query code from internal/db/queries.sql. |
make run / make run-daily |
Load .env and run the sync job (full / daily_sync). |
make local-db-up / make local-db-down |
Start / stop the local Docker Postgres. |
make local-run / make local-run-daily |
Run the sync against the local DB. |
make local-smoke / make local-smoke-daily |
local-db-up + run, in one command. |
make migrate / make migrate-down |
Apply pending migrations / roll back the most recent one out-of-band. |
make validate-fx ARGS='...' |
Run the FX validation CLI (see below). |
cmd/fx-validate reads rows from rates and compares them against ECB reference data.
make validate-fx ARGS='-date-from 2024-10-01 -date-to 2024-10-11 -output validation.csv'Output is a CSV with one row per checked observation, including absolute and relative diffs against the upstream value. Only -provider ECB is supported.
cmd/fx-migrate applies or rolls back the embedded migrations independently of the sync job — useful when you run with RUN_MIGRATIONS=false and gate schema changes behind a separate deploy step.
make migrate # apply all pending migrations
make migrate-down # roll back the most recent migrationIt reads the same DATABASE_URL / DB_* config as the job and accepts -env-file.
There is no internal scheduler — drive the binary from your scheduler of choice (k8s CronJob, ECS scheduled task, ordinary cron). A typical production schedule is daily_sync at 16:45 CET on weekdays, a few minutes after the ECB's daily publication window.
The fx_rates_run and fx_rates_provider_run log events (see docs/metrics.md) can drive alerting via Loki/Datadog/Vector.
fx-rates -version # prints the stamped build versionmake build injects git describe --tags --always --dirty. make release-build requires an explicit semver tag (e.g. VERSION=v0.1.0) and produces a reproducible static binary.
docs/DEPLOYMENT.md— container build, k8s CronJob runbook, exit codes, alerting, troubleshootingdocs/database-schema.md— table layouts, idempotency contract, useful audit queriesdocs/metrics.md— structured log event contract for dashboards and alertsdocs/providers.md— provider cataloguedocs/provider-credentials.md— credential expectations per providerdocs/currencies.md— supported ISO code list