Skip to content

feat(plugins): add lobsters column type - #27

Merged
aaronjmars merged 1 commit into
mainfrom
feat/lobsters-column
May 4, 2026
Merged

aaronjmars merged 1 commit into
mainfrom
feat/lobsters-column

Conversation

@aaronjmars

Copy link
Copy Markdown
Collaborator

What

34th column type — Lobsters (lobste.rs), a tech-focused community of ~30k engineers. Adjacent to Hacker News in audience but with narrower, higher-signal submissions and a tag taxonomy that supports filtered feeds (rust, ai, programming, security, …). Four modes: Hottest (front page), Newest, Active (sorted by recent comment activity), and Tag (single tag, or comma-separated multi-tag).

Why

Closes the obvious News & Web cluster gap — HN was the only general-tech discussion column. Lobsters fills the slot for users who want a second, smaller-signal stream alongside HN. Anyone running a founder dashboard / open-source maintainer dashboard / ML-research dashboard tends to read both — being able to stack `hacker-news` and `lobsters` next to each other on the same deck is the natural layout.

Why keyless: Lobsters exposes JSON variants of every public page — /hottest, /newest, /active, and per-tag /t/<tag> all return the same story array via .json. No auth, no token. Pagination flows through /page/N/{mode}.json (page 1 is the bare root, no /page/1/ segment — handled in endpointFor).

Changes

  • lib/integrations/lobsters.ts (new) — keyless JSON fetcher for the four modes, schema-drift-safe story mapper (drops rows missing short_id / title rather than rendering dead content), HTML strip for the description field (Lobsters returns sanitised fragments — p/br/code/em/a only — so a targeted regex strip is safe without pulling in a parser; falls back to description_plain if upstream sends it). Author unwrap handles both submitter_user: \"name\" and submitter_user: { username: \"name\" } shapes seen across recent API output. User-agent header identifies minitor — Lobsters' admins ask scrapers to identify themselves so they can throttle cooperatively if needed.
  • lib/columns/plugins/lobsters/{plugin,client,server}.ts (new) — standard 3-file plugin. Zod { mode: \"hottest\"|\"newest\"|\"active\"|\"tag\", tag: string }. Anchor icon (a nod to the lobster claw in lobste.rs branding without copying the proprietary logo). Brand red #ac130d (the lobster-claw red on lobste.rs/about) — distinct from HN's orange #ff6600 so the two news-source columns stay visually differentiated when stacked together.
  • lib/columns/plugins/manifest.ts, lib/columns/registry.ts, lib/columns/server-registry.ts — three registry edits per the README contract. Parity check at server module init throws if any registry drifts.
  • README.md — column count 33 → 34, News & Web row 4 → 5, hero paragraph picks up "Lobsters" alongside "Hacker News".

Lobsters-specific quirks handled

  • Tag mode supports comma-separated multi-tag filter (/t/rust,go.json) — matches Lobsters' upstream URL convention.
  • Tag mode with empty tag falls back to hottest in server.ts to avoid a 404 on /t/.json — column always renders something rather than throwing.
  • hasMore detection uses upstream page size (>=25) NOT post-filter slice, so the visible PAGE_SIZE clamp doesn't stop pagination early on full pages.
  • Tag pills in renderer — Lobsters' tag taxonomy is a key part of the signal; first 4 tags rendered as small pills under the snippet (HN's renderer doesn't have this since HN has no tags).
  • Comments URL distinct from external URL — story title links to the external article; comment-count badge links to lobste.rs/s/{short_id}. Matches HN's pattern.

What you get

No new env vars / build deps / schema migration — works the moment users add it, preserving the "first column up in under a minute, zero infra" README promise.


Built autonomously by Aeon

34th column type. Lobsters is a tech-focused community of ~30k engineers
running on lobste.rs — adjacent to Hacker News in audience but with
narrower, higher-signal submissions and a tag taxonomy that supports
filtered feeds (rust, ai, programming, security, ...).

Closes the obvious News & Web cluster gap — HN was the only general-tech
discussion column. Lobsters fills the slot for users who want a
second, smaller signal stream alongside HN.

Why keyless: Lobsters exposes JSON variants of every public page —
/hottest, /newest, /active, and per-tag /t/<tag> all return the same
story array via .json. No auth, no token. The story object shape is
documented at lobsters/lobsters @ app/views/stories/_story.json.jbuilder.
Pagination flows through /page/N/{mode}.json (page 1 is the bare root,
no /page/1/ segment — handled in endpointFor).

Standard 3-file plugin (plugin.ts / client.tsx / server.ts) plus the
3 registry edits (manifest, registry, server-registry — parity check
at server module init throws if drift). Same shape as hacker-news,
since the UX maps 1:1 (mode picker, optional filter input, score +
comment-count badges, comments URL distinct from external URL).

Lobsters-specific implementation details:
- Story object's submitter_user can be either a string username or an
  object with .username — unwrapAuthor handles both shapes.
- description field is sanitised HTML (p, br, code, em, a only); a
  targeted regex strip is safe without pulling in a parser. Falls back
  to description_plain if upstream sends it (rare).
- Tag mode supports comma-separated multi-tag filter (/t/rust,go.json) —
  matches Lobsters' upstream URL convention.
- Tag mode with empty tag falls back to hottest in server.ts to avoid a
  404 on /t/.json — column always renders something rather than throwing.
- hasMore detection uses upstream page size (>=25) NOT post-filter slice,
  so visible PAGE_SIZE clamp doesn't stop pagination early on full pages.
- Schema-drift safe: stories missing short_id or title are dropped rather
  than rendering as dead rows.
- User-agent header identifies minitor — Lobsters' admins ask scrapers
  to identify themselves; user-driven dashboard polling is well within
  their rate-limit guidance but a recognisable UA helps if they ever
  need to throttle cooperatively.

Anchor icon (a nod to the lobster anchor in lobste.rs branding without
copying the proprietary logo). Brand red #ac130d (the lobster-claw red
on lobste.rs/about) — distinct from HN's orange #ff6600 so the two
news-source columns stay visually differentiated when stacked together.

Renderer matches hacker-news layout (badge + author + relative time +
serif title + snippet + tag pills + score/comments footer). Tag pills
are unique to Lobsters — first 4 tags rendered as small pills under the
snippet, since Lobsters' tag taxonomy is a key part of the signal.

No new env vars / build deps / schema migration — works the moment
users add it, preserving the "first column up in under a minute, zero
infra" README promise. README updated: column count 33 → 34, News &
Web row 4 → 5, hero paragraph picks up "Lobsters" alongside "Hacker
News".
@aaronjmars
aaronjmars merged commit e28392d into main May 4, 2026
1 check passed
@aaronjmars
aaronjmars deleted the feat/lobsters-column branch May 4, 2026 12:53
aaronjmars added a commit that referenced this pull request Jul 9, 2026
Lockfile-only transitive bumps; no package.json changes needed.

- brace-expansion 5.0.5 -> 5.0.7  (GHSA-jxxr-4gwj-5jf2, #20, moderate)
    ReDoS-style DoS: large numeric range defeated the documented max guard.
    Affected copies under @ts-morph/common (shadcn) and @typescript-eslint.
- @babel/core   7.29.0 -> 7.29.7  (GHSA-4x5r-pxfx-6jf8, #27, low)
    Arbitrary file read via sourceMappingURL comment. Satisfies shadcn ^7.28.0.
- esbuild       0.27.7 -> 0.28.1  (GHSA-g7r4-m6w7-qqqr, #26, low, dev-only)
    Arbitrary file read via dev server on Windows. Pulled up by bumping the
    transitive tsx 4.21.0 -> 4.23.0 (drizzle-kit allows tsx ^4.21.0).

npm audit: 0 vulnerabilities. next build passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant