Cloudflare introduced a universal and egalitarian solution to the pluralistic problems inherent in distributed infrastructure.

Dubbed Meerkat, the web giant’s consensus service helps its internal services read and modify the same control-plane state from across its global base of 330-plus data centers. According to Cloudflare’s blog, such services “need guarantees that different readers never see inconsistent state and that the system remains available for writes even when some data centers or links fail.”

But with the internet prone to outages and queues, a consistent and globally available data system becomes hard to ensure. The Meerkat service works around this by employing a consensus algorithm by the name of QuePaxa, which like others of its ilk allows a set of machines to agree on the same sequence of values, as long as a majority remains functioning and able to communicate.

As it does not require a single authoritative leader service to ensure progress, QuePaxa allows all replicas or non-leaders to perform writes continuously even during what Cloudflare called the “tyranny of timeouts.”

Architecturally, a developer requests a cluster of Meerkat replicas, specifying which data centers may host the replicas, with Meerkat handling placement. A client sends an application-specific request to any replica, and the replica returns the application-specific response, as it turns application requests into events in a shared consensus log.

As such, every replica maintains the same sequence of decided log entries, though some replicas may temporarily lag behind and applications built on top of Meerkat, such as a key-value store or leasing system, read the log and build their own state from it.

“Because there is no required leader, the system never becomes unavailable or degraded due to a single replica … [and] there are no leader elections that degrade the system." Cloudflare explained. "And concurrent proposals made by different replicas constructively interfere … [which] is ideal for Cloudflare’s network, in which latencies can vary wildly."

The firm claimed to have run multiple proofs of concept with up to 50 globally distributed replicas, where leaders fail repeatedly but the cluster continues operating without an increase in error rate.

It added Meerkat is still not in production and will remain an internal-only service for the time being. It also stressed Meerkat has limitations as it wasn't designed to create general-purpose data systems like databases.

"All consensus algorithms come with a cost: lots of round-trips. ... If multiple replicas make proposals at the same time, it can take more. These communication costs point to the important performance limitation of consensus algorithms in general: proposal decision latency is proportional to the latency between some majority of replicas. So if your replicas are far from one another, latency will increase – there’s no getting around that."