Google Cloud got the ultimate test in July when its new customer Niantic released Pokémon GO, and the game shattered all prior estimates of player traffic. Engineers at Google and Niantic prepared for worst-case-scenario traffic at five times beyond expectations. Instead, they dealt with traffic 50 times beyond expectations.
Niantic’s engineering teams worked in concert with Google’s new Customer Reliability Engineering (CRE) group to cope with the unbelievable traffic.
Niantic had chosen Google Container Engine for its ability to orchestrate its container cluster at global scale to service millions of players as the game continuously adapted. “One of the more daring technical feats accomplished by Niantic and the Google CRE team was to upgrade to a newer version of Google Container Engine that would allow for more than a thousand additional nodes to be added to its container cluster,” writes Luke Stone, director of CRE in a Google corporate blog posting.
The teams cut over to the new version while millions of new players signed up and joined the Pokémon game.
On top of this upgrade, the engineers worked in concert to replace the network load balancer, deploying the newer and more sophisticated HTTP/S load balancer in its place. The HTTP/S load balancer is a global system tailored for HTTP/S traffic, offering higher throughput for the amount and types of traffic Pokémon GO was seeing.
The teams also used Google Cloud Datastore to provision extra capacity for Niantic to stay ahead of the record-setting growth.
A side benefit of the experience is that due to the scale of the container cluster and accompanying throughput, a multitude of bugs were identified, fixed, and merged into the Kubernetes open source project.
Google Makes Cloud a Top Priority
Just yesterday, at an event in San Francisco, Google announced its plans to elevate its cloud platform – now branded as Google Cloud. One of the new elements in Google Cloud is the Google Customer Reliability Engineering (CRE) team that played such an important role in the launch of Pokémon GO.
CRE is a new engagement model in which technical staff from Google integrate with customer teams, creating a shared responsibility for the success of cloud applications. Google said yesterday it plans to hire 1,000 people to help boost its cloud platform. And a big part of that is for CRE. It’s targeting a ratio of one Google engineer for every three enterprise customer engineers using Google Cloud.