Implementing Fencing Tokens for Distributed Transactions

A distributed lock is a lease with an expiry, not an unbreakable promise. The lock service (Redis, ZooKeeper, etcd) grants a lease for some duration, and the client is expected to renew it before it expires or finish its work within the window. The failure mode that breaks the exclusivity guarantee is any pause the lock service can't observe: a JVM garbage-collection stop-the-world pause, a suspended VM, a slow disk fsync, or a long network partition. From the lock service's point of view, a client that stops sending heartbeats looks identical to a dead client — so it does the only safe thing and expires the lease, handing the lock to the next requester.

The dangerous part is what happens next: the original client eventually resumes, has no idea time passed while it was paused, and proceeds to write to the resource as if it still held the lock — because, from its own perspective, it never released anything. Two writers, both believing they have exclusive access, is exactly the failure a lock was supposed to prevent.

A fencing token is a number issued by the lock service every time it grants the lock, guaranteed to increase monotonically across every grant — 33, then 34, then 35, regardless of which client receives each one. The client attaches this token to every write it sends to the protected resource. The insight that makes this work: instead of trying to prevent the paused client from writing (which the lock service cannot reliably do), you make the resource reject any write whose token is not higher than the highest one it has already seen.

In the inventory example, Client A acquires the lock and gets token 33, then pauses. The lock expires; Client B acquires it and gets token 34, writes successfully, and the storage records 34 as the last-seen token. Client A resumes, unaware time passed, and sends its write tagged with token 33. The storage layer compares 33 against its last-seen value of 34, sees that 33 is not greater, and rejects the write. Client A's stale write never lands — not because anyone stopped Client A from trying, but because the resource itself enforced the ordering.

The single most common mistake in a fencing-token implementation is putting the monotonicity check in the wrong place — inside the lock client's own logic, or in an application-layer guard that runs before the actual write. Neither is safe, because both can be skipped by the exact failure mode fencing tokens exist to handle: a client that is paused and has lost track of correct state. The check has meaning only if it happens at the boundary the paused client cannot bypass — the storage system that persists the write.

This is also why fencing tokens require the protected resource to support the check at all. If the resource is a plain key-value store with no way to attach a conditional write (compare-and-swap on a version field), you cannot retrofit fencing token safety onto it without adding that capability first. Systems like ZooKeeper (zxid), etcd (revision numbers), and a database row's version column all provide a natural place to store and compare the last-seen token.

Not every lock needs a fencing token. If the operation behind the lock is naturally idempotent (writing the same value twice is harmless) or if a duplicate write is cheap to detect and reconcile later, a plain lease-based lock without fencing is simpler and sufficient. Reach for fencing tokens specifically when the protected operation is a non-idempotent mutation with real consequences from double-application — inventory deltas, financial transfers, or any "apply this change exactly once" write where a duplicate corrupts state rather than merely repeating a no-op.

The realistic failure to keep in mind in production: a fencing token scheme that works in every test because tests don't reproduce multi-second GC pauses. Load-test with an artificial pause injected (a Thread.sleep or equivalent right after lock acquisition, before the write) to confirm the storage layer actually rejects the resulting stale write — a fencing token implementation that has never seen a real stale write in testing is unverified, not safe.