Monitor writer processes to handle silent process death - #81

Problem Statement

While the try/catch statement in CubDB.transaction/2 handles crashes and ordinary exits by a writer process, there are other scenarios where a writer process can die "silently" (e.g. OOM death, killed by an outside process, etc) that will not get caught by the safeguards within transaction/2.

In these cases, a writer process can die while still holding the write lock, preventing the write queue from advancing and leading to deadlock.

Proposed Solution

By having CubDB monitor a process as soon as it takes hold of the write lock, we are able to capture process deaths via a handle_info({:DOWN, ... callback that would not otherwise be caught and handled within transaction/2, allowing us to advance the write queue and free up the write lock. When the write lock is passed on in advance_write_queue/1, we can de-monitor the previous writer and monitor the new one (if applicable), ensuring only the active writer is being monitored.

Changes

• Adds :writer_ref to %CubDB.State{} to track the monitor reference for the writer process
• When a new :writer is assigned, call Process.monitor/1 on that writer and track the ref
• When advancing the write queue, call Process.demonitor/1 on the stored ref and monitor the new writer (if there is one)
• Adds a handle_info({:DOWN, ... for the active writer process to catch exits that bypass the safeguards in transaction/2, then advances the write queue
• Adds a new unit test covering the case of a "silent" writer death and validating it is fixed by new behavior