Insights collects popularity and usage metrics for open-source accounts and the things they publish — repositories, models, datasets, packages, containers, and services — and turns them into a historical REST API and a public dashboard.
Built for the ICICLE research ecosystem. The data model is platform-neutral, so it suits any community that wants a clearer picture of its open-source reach.
- One model across platforms. Accounts, repositories, models, datasets, packages, and containers share a consistent metric shape.
- Totals you can trust. Overlapping API windows are folded through daily watermarks, so nothing is counted twice — even when a job is retried.
- Durable collection. Valkey holds queued work while stateless workers scale independently of the HTTP service and the scheduler.
- Public by default, guarded where it matters. Reads need no credential; every write does.
- Credentials behind an interface. Jobs resolve platform tokens through a small provider contract rather than being coupled to one backend.
flowchart LR
U[Dashboard and API clients] --> APP[Vapor HTTP service]
APP --> DB[(PostgreSQL)]
S[Single scheduler] --> Q[(Valkey queues)]
Q --> W[One or more workers]
W --> GH[GitHub / Hugging Face]
W --> SP[SecretProvider]
W --> DB
| Component | Purpose | Scaling |
|---|---|---|
| HTTP service | Dashboard, REST API, OpenAPI | Scale freely |
| Queue worker | Claims and runs collection jobs | Scale freely |
| Scheduler | Evaluates the clocks and dispatches | Exactly one replica |
| PostgreSQL | Catalog, readings, due dates, watermarks | One managed database |
| Valkey | Queue storage and rate-limit counters | One shared service |
Two schedulers dispatch every due resource twice. That is the one hard scaling constraint.
| Platform | Metrics | Status |
|---|---|---|
| GitHub repositories | Stars, forks, subscribers, clones, views | Active |
| GitHub accounts | Followers | Active |
| Hugging Face | Likes, rolling downloads, lifetime downloads | Active |
| GHCR, npm, PyPI | — | Registered, not yet collected |
The scheduler scans hourly but each resource has its own cadence, seven days by default. See Collection schedule.
cp .env.example .envSet the Tapis values and DATABASE_TLS=disable, then:
just dbjust migratejust runThe full walkthrough, including the dashboard and the container stack, is in Local development.
Requires Swift 6.3+, Node 24+, just, and — for the container stack —
macOS 26 or later with Apple Container.
Administrators get an operations console at /admin: collection health, the catalog, vault
credential metadata, service tokens, and access control.
Start with Administering Insights.
Organised on Diátaxis, split by audience. The map is in docs/.
| Administrator | Developer | |
|---|---|---|
| Tutorial | Administering Insights | Local development |
| How-to | Deploy · Issue a token · Diagnose a failure | Add a collector · Dashboard toolchain · Run the tests |
| Reference | Admin console · Configuration · CLI | HTTP API · Data model · Invariants |
| Explanation | Watermarks · Authentication | Architecture · Decisions |
justLists every recipe, grouped: swift, web, cli, setup, containers, collect, stack.
See just recipes.
just test211 tests across 15 suites, run serially against a dedicated test database.
Set the five required variables, run migrations, create the signing keyset, and check the boot log. The full sequence is in Deploy Insights.
GNU General Public License v3.0. See LICENSE.
Developed as part of ICICLE (Intelligent Cyberinfrastructure with Computational Learning in the Environment), an NSF-funded AI institute (OAC 2112606).