The Linux Foundation’s Hyperledger project created a new tool to measure blockchain performance — Caliper. It allows users to measure the performance of a specific blockchain implementation using open source code.
Caliper was started in May 2017 and was designed and developed by Huawei, according to an email from a Hyperledger spokesperson.
Developers from other groups have since joined the work including contributors from Hyperchain, Oracle, Bitwise, Soramitsu, IBM, and the Budapest University of Technology and Economics. They’ve been collaborating within the Performance and Scalability Working Group (PSWG), which Hyperledger started in June 2017.
According to the Hyperledger project, there is not a commonly accepted blockchain benchmarking tool. There are some reporting mechanisms for blockchain implementations in various scenarios, but since they do not provide the source code, it is impossible to perform the same evaluation on different projects. Caliper addresses blockchain users’ desire for a tool that measures performance with agreed-upon metrics.
Caliper will produce reports containing a number of performance indicators such as transactions per second (TPS), transaction latency, and resource utilization. The tool is not intended to make judgments and will not publish benchmark results. And users should not claim a result is tested by Caliper unless the test environment is revealed. The tool references the definitions, metrics, and terminology as defined by the Hyperledger PWSG.
Caliper will be listed in the tools section of the Hyperledger website. In addition to tools, Hyperledger currently supports five blockchain sub projects. The technical steering committee of the Hyperledger project must approve each new sub project, and it’s looking for projects that “represent different thinking.”
Caliper relies on a functioning blockchain implementation as the benchmarking target. For now, three Hyperledger sub projects are in scope for Caliper: Fabric, Sawtooth, and Iroha. The hope is that the tool will provide benchmarking for many blockchain frameworks.
