Tags: Mu-L/marmot
Tags
Fix: DDL streaming replication - persist DDL to CDC storage Previously, DDL statements (CREATE/DROP/ALTER TABLE, CREATE/DROP INDEX) were stored only in write intents which were cleaned up after commit. This caused DDL transactions to stream with empty statements, breaking replication to followers. Changes: - Add OpTypeDDL constant for DDL operations in CDC - Add DDLSQL field to EncodedCapturedRow for DDL storage - Add writeDDLToCDC() to persist DDL to /cdc/raw/ during commit - Update sendChangeEvent() to handle OpTypeDDL with DDLChange payload - Fix gap detection in stream_client.go - txn_ids are HLC-based (nanoseconds), not sequential counters, so comparing against transaction count threshold caused infinite snapshot loops DDL replication flow: 1. Leader commit -> writeDDLToCDC() writes to /cdc/raw/ 2. Streaming reads from /cdc/raw/ via IterateCapturedRows() 3. sendChangeEvent() creates DDLChange payload for OpTypeDDL 4. Replica applies DDL via RewriteDDLForIdempotency() Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fix: CREATE DATABASE via write forwarding fails with syntax error The forward handler created ConnectionSession with TranspilationEnabled=false (the Go default). This caused the coordinator to skip transpilation when parsing forwarded SQL, which resulted in: - CREATE DATABASE being classified as StatementDDL instead of StatementCreateDatabase - Empty database name field (defaulting to session's current database) - SQLite receiving "CREATE DATABASE x" which it doesn't support The fix enables transpilation for forwarded queries, ensuring proper statement type detection for CREATE DATABASE, DROP DATABASE, and other special statements. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Feat: Add write forwarding from read-only replicas to leader Implements transparent write forwarding allowing read-only replicas to forward mutations to the leader node. Key features: - ForwardQuery gRPC endpoint for forwarding mutations and transactions - Transaction buffering with BEGIN/COMMIT/ROLLBACK support - Msgpack serialization for query parameters - TxnWaitQueue for waiting on transaction replication - DDL statements automatically wait for replication before returning - Multi-statement SQL support (e.g., CREATE TABLE with KEY generates both CREATE TABLE and CREATE INDEX) - Session state tracking for forwarded transactions Bug fixes included: - Transaction state synced to replica (ForwardedTxnActive field) - Multi-statement SQL properly split on leader - DDL schema visible immediately after forward completes - Fixed gossip integration test flakiness (gRPC warmup) New files: - grpc/forward_handler.go - Handles forwarded queries on leader - grpc/forward_session.go - Session management for forwarded connections - grpc/forward_params.go - Msgpack param serialization - replica/txn_wait_queue.go - Wait for transaction replication Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Feat: Wire up new admin endpoints in dashboard UI - Overview tab: Added cluster health banner with quorum status - Overview tab: Added databases table with file size, txn counts - Cluster tab: Extended with role, is_local, is_seed flags - New Replication tab: Shows peer replication lag per database - Transactions tab: Added aborted/all status filters and abort button - Database dropdown now dynamically populated from /admin/databases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fix: Replica snapshot verification and codebase cleanup Replica Snapshot Fix: - Filter out __marmot_system from snapshot file verification list - Replicas maintain their own independent system database - Master's StreamSnapshot already excludes system DB, but GetSnapshotInfo included it, causing verification mismatch Staticcheck Cleanup: - Remove unused functions: getConflictWindow, handleCreateDatabase, handleDropDatabase, syncPeer, parseIntentByTxnKey, fetchRow, readPIDFile, waitForRowCount, createTxnIDOrderingTestStore - Remove unused fields: db in PendingLocalExecution, inUpdateSet in deterministicChecker - Remove unused constants and delete db/cleanup.go (entirely dead code) - Fix unused metaStore assignment in grpc/server.go Deprecated API Migration: - Replace grpc.DialContext with grpc.NewClient (4 locations in catch_up.go) - Replace grpc.Dial with grpc.NewClient in gossip_integration_test.go - Replace grpc.DialContext with grpc.NewClient in stream_client.go - Remove grpc.WithBlock() option (not needed with NewClient) Code Quality: - Replace single-case select with simple channel receive - Remove redundant nil checks before len() on slices - Fix -0.0 literal (same as 0.0 in Go) - Remove unnecessary type assertion to same type - Fix unused append result in database_discovery_test.go - Run go mod tidy to remove unused dependencies Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Fix: CDC Type override breaking UPSERT replication Remove two lines that incorrectly override statement Type after ConvertToStatement() correctly computed it from CDC data. Root cause: For UPSERT (INSERT OR REPLACE) on existing rows, SQLite fires two hooks: DELETE (OldValues only) then INSERT (NewValues only). ConvertToStatement() correctly sets Type based on which values exist. The bug: Lines 441 and 897 were overriding Type back to REPLACE, causing the batch committer to call ApplyCDCInsert with nil NewValues for DELETE entries, resulting in "no values to insert" errors. Symptoms: - "partial commit: got 0 remote commit acks" errors - "ApplyCDCInsert: no values to insert" in batch committer - 99%+ of errors were UPSERT-related (9181/9204 in load test) The fix preserves SQL and Database for debugging but lets ConvertToStatement() determine the correct Type from CDC data. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Merge pull request maxpert#141 Improving Performance Multiple performance improvements
Merge pull request maxpert#139 from maxpert/feature/sqlite-batch-comm… …itter Introducing ability to batch commit changes at storage level
PreviousNext