Cockroach Labs doubled down on its testing methodology despite receiving mixed feedback from Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) alleging the company's test configurations weren't well suited to their strengths.

According to the report published late last year, GCP’s performance closed in on its larger rivals with the help of new and improved machine types. “GCP has made noticeable improvements in the TPC-C benchmark such that all three clouds fall within the same relative zone for top-end performance,” Cockroach Labs noted in a blog post last December.

While GCP had a notably standout performance, each cloud vendor had a category where they outshined the rest. Azure machines achieved significantly better results on the CPU micro benchmark; AWS won the combined storage read comparison across all categories with its i3en machine type; and GCP’s network throughput outperformed AWS and Azure.

Andy Woods, product manager at Cockroach Labs, explained in an interview with SDxCentral following the release of the initial report that it shared the results of its study with all three cloud providers prior to publication and incorporated their respective feedback to help improve its methodology and benchmarks.

“This is like a peer review process and a scientific paper,” Woods added. “We worked with them to get the most fair and accurate benchmarks possible.”

According to a more recent Cockroach Labs’ blog post, Amazon, Microsoft, and Google all shared similar sentiment with regards to how each cloud could have been tuned for optimal performance both on benchmarks and on real-world workloads.

For reference, Cockroach opted to test each cloud’s default machine configurations in the 2020 Cloud Report to “avoid biasing the results with claims that we better configured one cloud over another.” They believed testing default configurations for each benchmark would be a more accurate representation of performance considering “many cloud customers would not know how to configure machine types differently from the base defaults.”

“We stand by our decision to use default configurations in this experiment and welcome continued feedback from the clouds,” Cockroach Labs wrote. “Choosing how to configure your cloud is not intuitive and obviously many developers don’t have the benefit of speaking directly to the cloud providers to learn best in class performance configurations.”

Woods said in a follow up interview that Cockroach Labs will continue to test default machine configurations in its next report for CPU, network, storage, and TPC-C. He did indicate possible adjustments to testing methodology, and introducing new methods for the aforementioned factors and/or adjusting configuration flags.

While the cloud providers run many of the same benchmarking tests internally to measure and monitor improvements, each is still incentivized to show themselves off in the best possible light. That’s why, as Woods explained, “having somebody as a third party to make those [benchmarking] choices and apply them equally across all three clouds helps consumers have trust that they can actually make these comparisons fairly.”

And because these companies are constantly adopting new hardware, it’s important to note that cloud providers’ performance can change drastically and quickly as a result.

“Ultimately, each cloud can’t hide from public, open-source benchmarks,” Woods added. “As such, each cloud was eager to ensure that we had the best possible information to paint their cloud positively. And each cloud wanted to make sure we held the other clouds accountable for their statement.”