Repository navigation
Tags
Tags
Change data capture: change streams on the log service, cluster gate,… … reference consumer (#506) Change data capture: change streams, cluster gate, CDC consumer (#506) Change data capture for EloqDoc: $changeStream served from the log service instead of an oplog, a cluster gate that holds DDL still while a consumer copies the catalog, and a reference consumer that copies a cluster into another MongoDB and then follows its stream. Design in docs/cdc-change-stream.md, plan in docs/cdc-implementation-plan.md, operator runbook in docs/cdc-operations.md. The data substrate side is eloqdata/tx_service#577; the submodule points at its merged commit. Change streams - aggregate with $changeStream at collection, database or deployment level, then getMore: insert, replace and delete events with full documents; create, drop and index DDL events; collection-level invalidate. Updates are reported as replace (no updateDescription). - The server merges the log groups by commit timestamp before emitting. - Resume tokens and postBatchResumeToken on every reply, the opening one included; resumeAfter and startAfter resume exactly across pages, log groups and leader changes. The legacy shell's cursor tracks tokens, and the PyMongo, Node.js and Java drivers' watch() work unchanged. - New server parameters cdc_open_wait_ms (default 30 s) and cdc_scan_max_bytes. New error codes ChangeStreamStartNotReady (5000), ChangeStreamHistoryLost (286) and ChangeStreamFatalError (280). Cluster gate - Admin command {eloqClusterGate: {op: "pause" | "resume"}}, needing the new eloqClusterGate action (held by clusterManager, so clusterAdmin and root). While paused, every schema change fails with ClusterGatePaused (5001). Reference consumer - tools/cdc_consumer/ bootstraps an empty sink under the gate, then follows the deployment stream, keeping its position in the sink's admin.__eloq_cdc_progress. Fixes the work found outside CDC - KeyString decoded a decimal with an integer part above 2^53 one unit low. _id point lookups decoded numbers as int32. - Indexes under construction: updates left stale keys, a sparse build aborted the server on a document without the field, and multikey documents wrote only their first key. - Unique index builds over existing documents are refused (CannotCreateIndex): the tx service's build does not detect duplicates. - View definitions are read from system.views on every lookup, so a view change on one worker thread or node is visible to the others. - Write-conflict retries yield the coroutine instead of sleeping the brpc worker. - Startup skips a collection dropped between listing and lookup. - A create refused for its options (a validator in local, admin or config or on a system collection, or an unknown validationLevel or validationAction) no longer leaves its table behind; before, the server could not restart afterwards. This was a bug on main too. Tests and CI - New jstest suites eloq_cdc and eloq_fault (the latter only with the new WITH_FAULT_INJECT CMake option), runtime targets for the drivers, DDL recovery, the cluster gate and the consumer, and multi-node tests in cmake/tests/multinode/ (not run by CI). - The SCons jobs' jstest server runs with test commands enabled. Behavior changes to note: unique index builds on non-empty collections now fail; every write-conflict retry loop yields instead of sleeping; views are read per lookup. With txlog_cdc_retention_seconds at its default 0, a consumer behind the checkpoint when a purge runs gets ChangeStreamHistoryLost and must re-bootstrap. Verified: GitHub CI on this head (CMake and both SCons jobs: unit tests, all four jstest suites, TPC-C, drivers, DDL recovery, cluster gate, consumer); the multi-node tests, local and cloud mode, passed on a VM before the create-option fix.