Did you switch your Raft implementation & would you care to share your insights on the different implementations you guys have tried? (If there was a blog post on that, would appreciate the link).
This is the third Raft implementation for us. The original was goraft, then we attempted to make our own streaming raft implementation for high throughput.
This time around we realized we should only put very low write throughput stuff through Raft. And we wanted to pick on that has been out for a while and running in production in many places.
So we chose the Hashicorp implementation because we know them and know that it's been in production for a while. Combined with the fact that it can be backed by BoltDB, which we like.
My advice to you is to (a) model in TLA+. Writeup seems fine but, as you note, distribution is non-trivial (since intuition does not sufficiently/effectively inform the mental model). (2) do not rely on monotonic nanosecs from Go runtime. Only usec level precision is guaranteed monotonic. (3) if you don't own the client you are open to byzantine failure.
