Summary

This RFC proposes to use an index to trace leadership transfer, and if follower fails to start election, a response should be sent back to leader.

Motivation

When leader transfer leadership to a follower, follower may and may not start an election. Leader can't know precisely what happen, so it will stop read/write and wait for an election timeout and then try to retain leadership if no one campaigns. We observe some unexpected high latency when a leadership transfer fails. Note that failure doesn't have to be caused by network failure, it can also be caused by slow apply of logs. For example, a newly promoted voter may not start campaign if conf change is applied locally.

Detailed design

We can introduce an index to trace every leadership transfer. Everytime leadership transfer happens, index should increase by 1. The index is also sent via transfer command. If a follower checks its own state, and decides not to campaign, it should send back a TransferLeaderResponse to tell leader its decision. Leader finds a rejected response's index matches its own latest transfer index, then abort leadership transfer immediately.

Unresolved questions

What if transfer command is dropped due to network failure? It may be hard to handle all situations, but at lease should make it work as expected when infrastructures work as expected.