Skip to content

Tags

Tags

nightly

Toggle nightly's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Change data capture: change streams on the log service, cluster gate,…

… reference consumer (#506)

Change data capture: change streams, cluster gate, CDC consumer (#506)

Change data capture for EloqDoc: $changeStream served from the log
service instead of an oplog, a cluster gate that holds DDL still while a
consumer copies the catalog, and a reference consumer that copies a
cluster into another MongoDB and then follows its stream. Design in
docs/cdc-change-stream.md, plan in docs/cdc-implementation-plan.md,
operator runbook in docs/cdc-operations.md. The data substrate side is
eloqdata/tx_service#577; the submodule points at its merged commit.

Change streams
- aggregate with $changeStream at collection, database or deployment
  level, then getMore: insert, replace and delete events with full
  documents; create, drop and index DDL events; collection-level
  invalidate. Updates are reported as replace (no updateDescription).
- The server merges the log groups by commit timestamp before emitting.
- Resume tokens and postBatchResumeToken on every reply, the opening
  one included; resumeAfter and startAfter resume exactly across pages,
  log groups and leader changes. The legacy shell's cursor tracks
  tokens, and the PyMongo, Node.js and Java drivers' watch() work
  unchanged.
- New server parameters cdc_open_wait_ms (default 30 s) and
  cdc_scan_max_bytes. New error codes ChangeStreamStartNotReady (5000),
  ChangeStreamHistoryLost (286) and ChangeStreamFatalError (280).

Cluster gate
- Admin command {eloqClusterGate: {op: "pause" | "resume"}}, needing the
  new eloqClusterGate action (held by clusterManager, so clusterAdmin
  and root). While paused, every schema change fails with
  ClusterGatePaused (5001).

Reference consumer
- tools/cdc_consumer/ bootstraps an empty sink under the gate, then
  follows the deployment stream, keeping its position in the sink's
  admin.__eloq_cdc_progress.

Fixes the work found outside CDC
- KeyString decoded a decimal with an integer part above 2^53 one unit
  low. _id point lookups decoded numbers as int32.
- Indexes under construction: updates left stale keys, a sparse build
  aborted the server on a document without the field, and multikey
  documents wrote only their first key.
- Unique index builds over existing documents are refused
  (CannotCreateIndex): the tx service's build does not detect
  duplicates.
- View definitions are read from system.views on every lookup, so a view
  change on one worker thread or node is visible to the others.
- Write-conflict retries yield the coroutine instead of sleeping the
  brpc worker.
- Startup skips a collection dropped between listing and lookup.
- A create refused for its options (a validator in local, admin or
  config or on a system collection, or an unknown validationLevel or
  validationAction) no longer leaves its table behind; before, the
  server could not restart afterwards. This was a bug on main too.

Tests and CI
- New jstest suites eloq_cdc and eloq_fault (the latter only with the
  new WITH_FAULT_INJECT CMake option), runtime targets for the drivers,
  DDL recovery, the cluster gate and the consumer, and multi-node tests
  in cmake/tests/multinode/ (not run by CI).
- The SCons jobs' jstest server runs with test commands enabled.

Behavior changes to note: unique index builds on non-empty collections
now fail; every write-conflict retry loop yields instead of sleeping;
views are read per lookup. With txlog_cdc_retention_seconds at its
default 0, a consumer behind the checkpoint when a purge runs gets
ChangeStreamHistoryLost and must re-bootstrap.

Verified: GitHub CI on this head (CMake and both SCons jobs: unit
tests, all four jstest suites, TPC-C, drivers, DDL recovery, cluster
gate, consumer); the multi-node tests, local and cloud mode, passed on
a VM before the create-option fix.

0.3.0

Toggle 0.3.0's commit message
rename s3 path

0.2.8

Toggle 0.2.8's commit message
0.2.8 release

0.2.7

Toggle 0.2.7's commit message
mongo::AuthorizationManagerImpl::acquireUser: Fix thread group thread…

… blocked on pthread condition_variable (#293)

0.2.6

Toggle 0.2.6's commit message
Revert coroutine stack (#236)

This PR reverts the use of boost::context::protected_fixedsize_stack, because
memory is not reused even though it is a class member variable. Each callcc will
allocate memory by calling mmap/mprotect.

0.2.5

Toggle 0.2.5's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Disable test queryoptimizer3.js (#186)

0.2.4

Toggle 0.2.4's commit message
chore: fix TAGGED flag condition