This blog was written by Samuel Dare, Blockchain Engineer at Brave
Marching towards Brave’s mission to create a decentralised ecosystem for digital advertising that respects user’s privacy, we aim to incrementally decentralize components of our architecture as they relate to the Basic Attention Token (BAT) as well as Brave’s advertiser and publisher ecosystem. We also aim to maintain or improve the performance, security and privacy guarantees which the BAT ecosystem currently offers.
Public blockchains (like the Ethereum Mainnet) are open networks, allowing anyone to both add and verify data in the network. This guarantees a high level of decentralization2, non-repudiation3, and censorship resistance at the expense of :
- Transaction Throughput: In its current implementation, Ethereum nodes broadcast transactions at an average of 15 Transactions Per Second (TPS). This throughput pales in comparison to VISA4, which claims a performance of 65,000+ TPS . This low transaction speed is because (i) there are many nodes taking part in consensus (ii) it takes a longer time to broadcast the transactions to all the nodes in the network.
- Execution Costs: To solve the halting problem5, Turing complete public blockchains attach a monetary cost (gas) to the execution of commands. This cost is undesirable and can be unpredictable.
- Lack of Privacy: Confidentiality and Anonymity are also issues, as participants are privy to all the data in the network and interactions between participants.
Permissioned blockchains, on the other hand, have transaction throughputs orders of magnitude higher than their public counterparts while potentially improving privacy. Permissioned blockchains benefit from less expensive consensus algorithms and lower communication costs. Permissioned consortiums can be broader and more open than mining consortiums if they’re chosen carefully: ultimately we think a consortium can be as decentralized and censorship resistant, or even more so than mining consortiums or proof of stake schemes, which all potentially suffer from oligarchic problems. Based on this, we consider permissioned blockchains a reasonable compromise when choosing building blocks for the future Apollo phase of the BAT ecosystem.
While Ethereum Clients are a core component of Web 3.0, and there are many such clients already developed, the lack of standardisation across them makes it difficult to test the technology.
To remedy this and enable testing on a level playing field, we created Mjolnir. Mjolnir is a tool for deploying and benchmarking Permissioned Ethereum Blockchain implementations easily. We hope that other teams can benefit from the results and use the tool for their testing.
Ethereum Permissioned Blockchains To Compare
Consensus algorithms are protocols used to achieve agreement on data, from a single value up to the complete current state of a distributed system, among distributed processes or systems. They are primarily used to achieve reliability in a network involving multiple distributed nodes that contain the same information. In the case of permissioned blockchains we find that consensus algorithms are often a bottleneck6. Keeping communications costs (and number of nodes) fixed, consensus algorithms have different assumptions and communication models which might constitute an overhead.
To assess the effectiveness of our tool, we evaluate Mjolnir using two permissioned blockchains:
- Quorum7 running Istanbul Byzantine Fault Tolerance (IBFT): Quorum is a fork of the Go-Ethereum client that has been modified to facilitate private transactions. IBFT is an implementation of Practical Byzantine Fault Tolerance that requires validators to vote on blocks through multiple rounds.
- PoA and lab10 Coop’s fork of the Parity Ethereum Client, which runs on the Honey Badger Byzantine Fault Tolerance (HBBFT) algorithm. HBBFT promises sub section block finality8, asynchronous safety and resistance to selective censorship.
While Quorum has the benefit of being the most deployed permissioned client, HBBFT’s finality, asynchronous safety, and selective-censorship resistance are the properties we believe help abate some commonly expressed concerns over the use of permissioned systems.
While in our experiments, Quorum outperformed with regards to throughput, HBBFT fared better when faced with clock skew and had a better memory profile, consuming less memory for every new node added to the cluster.
In the remainder of this blog post, we’ll first dive into the system design of Mjolnir and how to use it to deploy different configurations of permissioned blockchain. Finally, we’ll outline some performance results that compare Quorum and Parity HBBFT blockchains under different configurations and deployed using Mjolnir.
Mjolnir: System Design
The Mjolnir system can be grouped into three components (Figure 1): (1) the deployment, (2) testing, and (3) monitoring components. Each component is divided into subcomponents, so let’s dive into each of them:
Figure 1. Overview of Mjolnir’s components
Component 1: Deploy
These components help configure, validate, and build the nodes. They leverage Terraform and Amazon Web Services and bundle all functionality in a CLI. Let’s see what each of the subcomponents do:
CLI: The CLI provides a user-friendly entry point to the Mjolnir toolkit. The user needs only to update a YAML file with their desired settings and execute a subcommand with the YAML file. The CLI instructs Terraform to deploy the requisite infrastructure to the target environment, which in our case is AWS. In addition to this, it also acts as a management console for the cluster.
Figure 2. Sample Configuration File
Terraform: Terraform is a tool for declaratively managing cloud-based infrastructure and service by way of the Infrastructure as Code paradigm. Being declarative means that it builds according to a target state, making it easier to reason about the codebase.
Amazon Web Services (AWS): As mentioned above, our target environment is AWS. Services currently employed include (but are not limited to):
- Elastic Container Service (ECS): ECS is a fully managed containerisation service. While one could argue that we take a performance hit by not choosing to go barebones, we run the service in EC2 mode which allows for some optimizations, while still affording the benefits of a managed service. The nodes in the network are deployed to ECS.
- Simple Queue Service (SQS): A fully managed message queueing service that allows the decoupling of services and scaling of architecture. In our setup, nodes are laid out as microservices. This allowed us to pass different runtime parameters to our containers (i.e. clock skew, chaos testing)
- Simple Storage Service (S3): This provided both ephemeral and long term storage to the clusters. Ephemeral storage was important when bringing up the service as declarative languages do not readily lend themselves to programmability. Given the above, s3 was a good staging area.
Component 2: Test
These components are responsible for initiating and collecting results on the tests suite. They include Chain hammer, libfaketime and pumba:
Chain hammer: This is a collection of python scripts that rapidly sends asynchronous transactions to an Ethereum node.
libfaketime: This library allows us to intercept system calls made within the containers and return values modified by the user. With this, we were able to simulate clock skew between the containers.
Pumba: This is a tool for Chaos testing and network emulation within containers. While it’s functionality doesn’t quite match Netflix’s Chaos Monkey, we found it suitable for simulating both partial synchrony and asynchrony within the containers running the Ethereum nodes.
Component 3: Monitor
The monitoring component collects and shows data from the tests.
Grafana: Collects time series data and presents it in dashboards which help to visualize and analyze how the Ethereum nodes perform over time. The dashboards enable us to understand any exceptional node and network behaviour.
Loki/Prometheus: We use log aggregation systems which are responsible for steaming logs and metrics from the ECS instances to Grafana.
Experiments
In order to show-case Mjolnir’s features, we set to compare the performance of a Quorum-based blockchain with a Parity HBBFT-based blockchain. In order to do so, we leverage the components described in the section above to spin up, orchestrate, and monitor the cluster using Mjolnir.
Client Configuration
The table below outlines the different configurations of the tested clients. Both clusters were deployed on AWS with each validator node running on its own EC2 instance.
| Client | Quorum | Parity HBBFT |
|---|---|---|
| Version | v2.2.5 | hbbft-branch |
| Commit Hash | 99f7fd6733a93ee7619d1c740e0d4cd7643b6700 | 50c4e62b93df9652df1f614a5ca36438ecceb026 |
| EC2 Spec | t.2xLarge (8 vCPUs, 32 GB) | |
| AWS Region | us-east-2 | |
| Number of Nodes | 4 | |
| Gas Limit | 800,000,000 | |
Transactions Per Second (TPS)
One important metric is the transactions per second that the blockchain can achieve. Often a functional requirement in most Blockchain systems, TPS is a proxy for throughput as it measures how fast the client can reach consensus on the next block and propagate it through the network. In our experiments, while Quorum achieves a max TPS of 890, HBBFT reaches 790 TPS.
| Number of Nodes | 4 | |
|---|---|---|
| Client | Quorum | HBBFT |
| Max TPS | 890 | 790 |
| Average TPS | 730 | 645 |
Resistance to Clock Skew
A large number of consensus mechanisms make strong synchrony assumptions, which render the protocols fragile to discrepancies between the system clocks on the nodes in the cluster. It is important to note that this refers to changing the time in the containers and not the host VM. Testing the effect of clock skew on the host is outside the scope of this exercise.
| Client | Quorum | HBBFT |
|---|---|---|
| Result | The nodes disconnect from the network with the introduction of a 1 second clock skew. | Seems to continue to operate regardless of clock skew. |
Memory Utilization
We measure the increase in the memory consumption of the clients as we increase the number of nodes in the consortium. This is a proxy for communication complexity, as we imagine that the nodes have to do more work to broadcast messages.
| Client | Quorum | HBBFT |
|---|---|---|
| Result | Quorum seems to perform worse. As can be seen from the graph, its memory consumption is “more than quadratic” for every new node added to the consortium. | Parity seems to scale quadratically with regards to both communication and memory costs. |
Summary
- Quorum achieves a max TPS of 890, while HBBFT reaches 790 TPS.
- HBBFT is resistant to clock skew.
- HBBFT consumes less memory for every new node added to the network.
Conclusion
In this blog post, we present Mjolnir: a tool for developers to easily deploy and test different Ethereum Permissioned Blockchain implementations. We outline the architecture of Mjolnir and how it can help with comparing different permissioned blockchain clients. In addition to presenting Mjolnir, we used it to deploy and compare Quorum against HBBFT based blockchains. Overall, both implementations seem promising, with Quorum leading the race in throughput, and HBBFT presenting an innovation in permissioned consensus. We would like to see further experiments with regards to the geographic distribution of nodes and sending transactions round-robin style to different nodes within the consortium, as we believe it aligns closer to real scenarios. We are providing Mjolnir open-sourced and publicly available, aiming to help the community to iterate through their blockchain ideation/deployment process quicker. There is still more to be done, therefore we call on contributors to help both maintain and augment Mjolnir’s capabilities.
References / Notes:
- BAT Roadmap: https://basicattentiontoken.org/bat-roadmap-1-0/
- Vitalik Buterin: Meaning of Decentralisation
- Non-Repudiation: https://en.wikipedia.org/wiki/Non-repudiation
- VISA Factsheet: About Visa Fact Sheet
- Halting Problem: Halting problem
- This assumes that implementations of the protocol contain no flaws
- Go Quorum: https://www.goquorum.com/
- Finality (Binance Academy): https://www.binance.vision/glossary/finality
Further Reading:
- Consensus Algorithms: IBFT or Raft: https://kaleido.io/consensus-algorithms-poa-ibft-or-raft/
- Parity Ethereum Client: https://www.parity.io/ethereum/
- BuildingHoneyBadgerBFT: https://medium.com/poa-network/poa-network-buiilding-honey-badger-dft-c953afa4d926
- Miller et al: The Honey Badger of BFT protocol https://eprint.iacr.org/2016/199
