You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The ingester checkpoints (advances LSN) after writing to the OS pipe, not after the output provider confirms delivery to the destination (e.g., ASB). If the output provider crashes between receiving data and delivering it, the data is lost — the ingester thinks it was delivered.
This is a non-negotiable design requirement: a payload must not be marked as saved without confirmation from the output provider.
Current Flow (broken)
Ingester writes to stdout (pipe) → checkpoint LSN ← TOO EARLY
↓
CLI relays to output provider stdin
↓
Output provider sends to ASB ← confirmation never flows back
Proposed Flow
Ingester writes to stdout (pipe) → does NOT checkpoint yet
↓
CLI relays to output provider stdin
↓
Output provider sends to ASB → writes ack to stdout: {"ack":N}
↓
CLI relays ack back to ingester stdin
↓
Ingester receives ack → NOW checkpoint LSN
Design Considerations
Ingester must read acks from stdin — currently it closes stdin after receiving config. Needs to keep it open and read ack lines.
Output provider writes acks to stdout — after each batch is confirmed by ASB, emit {"ack": <count>} or {"ack_lsn": "<lsn>"}.
CLI relays acks — output provider stdout → ingester stdin. The CLI already reads output provider stdout (for forwarding to os.Stdout). It needs to detect ack lines and route them back to the ingester.
Problem
The ingester checkpoints (advances LSN) after writing to the OS pipe, not after the output provider confirms delivery to the destination (e.g., ASB). If the output provider crashes between receiving data and delivering it, the data is lost — the ingester thinks it was delivered.
This is a non-negotiable design requirement: a payload must not be marked as saved without confirmation from the output provider.
Current Flow (broken)
Proposed Flow
Design Considerations
{"ack": <count>}or{"ack_lsn": "<lsn>"}.Impact
dstreamCLI (relay acks),dstream-ingester-mssql(read acks, defer checkpoint),dstream-out-asb(emit acks),dstream-sdk-dotnet(ack support in SDK)ackmessage type to stdoutChangePublisherinterface in the ingester needs to change —PublishChangesshould not return done until the ack is receivedAcceptance Criteria
SendMessagesAsyncsucceeds)