Lighthouse — A Configurable Multi-Source Research OS, Orchestrated by Kestra

One engine. Four topic profiles. Daily research briefs to Notion, Slack, Discord, and email. Chat-the-brief and on-demand deep-dives via Kestra Apps. Drop a YAML topic file in, get a personalised brief tomorrow.

Live demo (Next.js front door): demo-beta-topaz.vercel.app

Agent skill — use Lighthouse (not develop it): connects agents via WebMCP on the live dashboard.

npx skills add guglxni/lighthouse-kestra --skill lighthouse -y

See skills/README.md for install flags and WebMCP tool list.

Lighthouse is a research operating system built entirely on Kestra. It ingests RSS, arxiv, GitHub trending, HN/Reddit, transcribed YouTube/podcasts, and any arbitrary web URL; embeds and de-duplicates everything in pgvector; classifies and summarises through LiteLLM (OpenAI-compatible baseUrl) with optional multi-model / Azure Kimi fallbacks (see CONVENTIONS.md); and delivers a daily brief to your channels. Everything is configured via a single YAML topic profile, so spinning up a brief for a new topic takes one file — no code changes.

LLM / BYOK: Models are not locked to a single vendor. Kestra talks OpenAI-compatible HTTP to LiteLLM (or any gateway you point LITELLM_BASE_URL at). You define providers and aliases in infra/litellm/config.yaml. See docs/BYOK.md (includes Exa BYOK for semantic search).

Real ingest, no mocks: All seven core source types call live APIs/feeds and persist to Postgres (see docs/INGEST_SOURCES.md). Heroku / student credits: see docs/DEPLOY_HEROKU.md — the default stack targets Docker Compose; full Heroku parity is non-trivial.

Four topic profiles ship preconfigured:

• agentic-eng — Anthropic / OpenAI / Cursor / MCP / agent frameworks
• solana-zk — Solana protocol + ZK proof systems + audits
• indie-saas — Bootstrapped SaaS, distribution, growth case studies
• data-eng-ai — Lakehouse, orchestration, vector DBs, RAG infra

git clone https://github.com/guglxni/lighthouse-kestra.git
cd lighthouse-kestra
cp .env.example .env       # fill LiteLLM + provider keys (see table)
docker compose -f infra/docker-compose.yml up -d
open http://localhost:8080 # Kestra UI; Apps live under /ui/apps

Required secrets (set in .env or your secret manager):

Full list and default values in .env.example.

flowchart LR
    subgraph sources [Sources]
        rss[RSS via Miniflux]
        arxiv[arxiv API]
        ghtrend[GitHub trending]
        hn[HN Algolia + Reddit JSON]
        yt[YouTube + Podcasts]
        web[Web articles]
    end

    subgraph ingest [Ingest namespace]
        ingrss[lighthouse.ingest.rss]
        ingarxiv[lighthouse.ingest.arxiv]
        inggh[lighthouse.ingest.github_trending]
        inghn[lighthouse.ingest.hn_reddit]
        ingav[lighthouse.ingest.audio_video]
        ingweb[lighthouse.ingest.web_articles]
    end

    subgraph process [Process namespace]
        embed[embed_dedup<br/>pgvector]
        classify[classify<br/>multi-LLM fallback]
        cluster[cluster_summarize<br/>BERTopic + AI Agent]
    end

    subgraph deliver [Deliver namespace]
        brief[deliver.brief<br/>Notion + Slack + Discord + email]
        chat[serve.chat_brief<br/>Kestra App]
        deep[serve.deepdive<br/>GPT-Researcher in Docker]
    end

    subgraph monitor [Monitors namespace]
        alert[monitors.alerts<br/>single Slack pager]
        gc[maintenance.gc<br/>archive + prune]
    end

    rss --> ingrss
    arxiv --> ingarxiv
    ghtrend --> inggh
    hn --> inghn
    yt --> ingav
    web --> ingweb

    ingrss & ingarxiv & inggh & inghn & ingav & ingweb --> embed
    embed --> classify --> cluster --> brief
    cluster --> chat
    cluster --> deep

    ingest -. FAILED/WARNING .-> alert
    process -. FAILED/WARNING .-> alert
    deliver -. FAILED/WARNING .-> alert

Deeper dive in ARCHITECTURE.md. Ingest source inventory (all live APIs / feeds): docs/INGEST_SOURCES.md.

Lighthouse uses Graphify to turn the canonical docs plus a generated flow index into a browsable knowledge graph (graph.json, interactive HTML, GRAPH_REPORT.md). That helps contributors see how namespaces, ingest → process → deliver paths, LiteLLM BYOK, and data/PG concepts connect without re-reading every YAML file.

One command (from repo root, with an LLM key set for markdown semantic extraction — default backend openai):

./scripts/graphify_lighthouse.sh
# or
make graphify

Outputs land in docs/graphify/output/ (gitignored if large). The script copies graph.html → report.html for a stable name. Optional call-flow HTML: GRAPHIFY_CALLFLOW=1 ./scripts/graphify_lighthouse.sh.

Install (pick one) — Graphify’s PyPI package is graphifyy (CLI remains graphify); use 0.7+ for graphify extract:

uv tool install graphifyy && graphify install
# or: pipx install graphifyy && graphify install

If your globally installed graphify is older, the script falls back to uvx --from 'graphifyy>=0.7.13' when uv is available. Set GRAPHIFY_BACKEND (e.g. gemini, claude) and the matching API key per Graphify’s README.

Groq (full repo graphify extract): Graphify has no dedicated groq backend, but Groq’s HTTP API is OpenAI-compatible. Use the ollama backend (same OpenAI client) with Groq’s base URL and a current API key from Groq Console. The openai Python package must be available (uvx --with openai bundles it).

Graphify’s ollama backend sends Ollama-only fields (keep_alive, large max_completion_tokens) that Groq rejects. Use the repo helper scripts/graphify_extract_groq.py, which patches those calls for groq.com URLs. Many Groq models cap max_completion_tokens at 8192 — set GRAPHIFY_MAX_OUTPUT_TOKENS=8192 or extraction may return HTTP 400.

export OLLAMA_BASE_URL="https://api.groq.com/openai/v1"
export OLLAMA_API_KEY="your_groq_key"   # starts with gsk_
export GRAPHIFY_MAX_OUTPUT_TOKENS=8192
uvx --with openai --from graphifyy python scripts/graphify_extract_groq.py extract . \
  --backend ollama \
  --model "meta-llama/llama-4-scout-17b-16e-instruct" \
  --out .

Outputs appear under ./graphify-out/ (graph.json, graph.html, GRAPH_REPORT.md). If semantic chunks fail, confirm the key is valid (401 / expired_api_key means rotate the key in Groq), that uvx includes --with openai, and that output token cap matches the model.

Submodule (optional): vendor Graphify next to this repo for skills/hooks or local patches (use on case-sensitive filesystems; default macOS APFS is often case-insensitive and upstream may appear permanently dirty):

git submodule add https://github.com/safishamsi/graphify graphify
git submodule update --init --recursive

CI does not run Graphify by default (LLM keys); refresh graphs on release or when architecture docs change.

lighthouse/
  flows/
    ingest/        rss arxiv github_trending hn_reddit audio_video web_articles
    process/       embed_dedup classify cluster_summarize
    deliver/       brief
    serve/         chat_brief deepdive
    apps/          dashboard chat deepdive
    monitors/      alerts gc
    maintenance/   graphify_docs (docs reminder)
    _namespace_files/
      topics/      agentic-eng solana-zk indie-saas data-eng-ai
      prompts/     system.md
  scripts/         transcribe.py extract_trafilatura.py bertopic_cluster.py deepdive_runner.py
  sql/             pgvector_init.sql schema.sql
  tests/flow_tests/test_rss_dedup.yaml test_classify_schema.yaml test_brief_shape.yaml
  infra/           docker-compose.yml Dockerfile.worker
  blueprint/       lighthouse-research-quickstart.yaml PR_BODY.md
  .github/workflows/validate-flows.yml
  README.md ARCHITECTURE.md CONVENTIONS.md docs/ SECURITY.md scripts/graphify_lighthouse.sh
  docs/TODO.md LICENSE

Optional: add images under docs/img/ after your first successful run (dashboard, chat, brief in Slack/Notion, etc.) and reference them here.

See SECURITY.md for the threat model (Docker socket, unauthenticated UI, webhook URLs, dependency pins) and how to report issues.

PRs welcome — adding a topic profile is the easiest first contribution: drop a new file in flows/_namespace_files/topics/<slug>.yaml, add a Discord webhook secret if you want Discord delivery, and the existing flows pick it up.

For larger changes, please open an issue first to align on shape.

MIT — see LICENSE.