Overview
Blockchain presents unique challenges to I/O and storage processing. Circuit Blvd has been working with Coinplug to enhance Coinplug's blockchain platform Metadium. Metadium is based on Ethereum but in terms of performance, its optimization features result in more than 10x higher Transactions Per Second (TPS). However, Coinplug started to observe I/O bottlenecks when overall TPS has been enhanced from Ethereum.
The Storage Architecture of Metadium
Metadium's storage architecture inherits Ethereum's which has been analyzed by a few academic publications. This 2018 paper provides a good overview of how Ethereum's internal storage works and where potential problems could occur. Ethereum stores most of the internal data as a key-value pair. The keys are 256-bit keccak cryptographic hash. The key-value storage is organized as multiple Merkle Patricia Tries (MPT) where each trie stores certain categorical data such as transactions and block headers. Leaf nodes of MPT stores the data while non-leaf nodes store up to 16 pointers to child nodes. MPT nodes are stored in LevelDB key-value store.
The following diagram illustrates how an MPT tree is traversed to obtain the data at leaf nodes. The most significant nibble (4 bits) of the input cryptographic hash (0x5) takes us to the child node FN1,5. As one traverses one level down, they follow the next hex digit in the cryptographic hash key. Therefore, one needs to traverse as many as 64 levels (256 bit divided by 4 bit for each hex digit) to reach the target leaf node. Now, we can foresee the potential performance problem here. For a single data value in an MPT, one needs as many as 64 LevelDB get() operations. In addition, this is further amplified by internal read amplifications of LevelDB as well as the underlying file system and storage device.
This page in Ethereum github wiki gives definitive details on how MPTs work. The paper mentioned above claims that over 93% of total processing time is spent in storage processing when I/O heavy workloads are applied to an Ethereum node.
Installation of Metadium on on-premise and on AWS servers
We present necessary steps to prepare Metadium to measure storage performance both on local and AWS servers. We would like to run I/O intensive workloads over different types of key-value stores and SSDs. This Metadium github page gives a complete guideline to get Metadium up and running for your reference. To install Metadium on a local server, we start with pre-requisites:
golang version >= 1.10 is required.
cmake, numactl-devel, tbb-devel, jemalloc, zstd, snappy, lz4 and maybe others are required
In some machines, -zstd -lbz2 -llz4 -lsnappy -ljemalloc may not be needed. Remove them in Makefile.
Git clone from https://github.com/METADIUM/go-metadium.git. Build by doing 'make' for all targets. After the build, look at build/ directory to find metadium.tar.gz. The tar file should include:
bin/gmet
bin/gmet.sh
bin/solc.sh
bin/logrot
conf/MetadiumGovernance.js
conf/genesis-template.json
conf/config.json.exampleNow we need to set up a new network for our performance profiling work. Let's use gmet.sh script which assumes Metadium data directory is /opt/<node-name>. Let's name the <node-name> 'meta' in our example. Unpack the tar file.
Next, we are ready to create the config file by editing conf/config.json.example and save it as config.json. Do the following to create account and node files.
bin/gmet metadium new-account --out <account-file>
bin/gmet metadium new-nodekey --out <node-key-file>
bin/gmet metadium nodeid <node-key-file>Let's start the node by copying necessary files into the right places.
mkdir /opt/meta
cd /opt/meta
mkdir geth
cp <node-key-file> geth/nodekey
mkdir keystore
chmod 0700 keystore
cp <account-file> keystore/
bin/gmet.sh init meta config.json 8588 # genesis.json gets generated # compile/load governance contract
bin/solc.sh -p 1 MetadiumAdmin.sol MetadiumAdmin.js
bin/gmet.sh start #start the nodeNow we can create governance contract and ready to apply application workloads.
bin/gmet.sh console
... > loadScript('MetadiumAdmin.js')
> personal.unlockAccount(<address>, <password>, <duration-in-second>)
> aa = AdminAnchor_new(); # wait for the completion, i.e. the address of AdminAnchor contract.
> ad = Admin_new();
# wait for the completion, i.e. the address of Admin contract.
> ad.switchAdmin(aa.address, ad.address, {from:eth.accounts[0], gas:1000000});Installation on AWS servers is basically the same as installation for on-premise servers. We have created an EC2 instance out of type t3.xlarge and installed CentOS 7. We have experimented Ubuntu also but obtained almost the same performance numbers.
In the next technote...
In the next technote, we will provide initial results of Circuit Blvd's GriffinDB performance on Metadium platform in comparison with the default internal DB, RocksDB. Coinplug selected RocksDB as the default internal DB over Ethereum's LevelDB after it verified that RocksDB brings about 2x TPS increase just by replacing LevelDB. GriffinDB is optimized to highly randomized point queries by utilizing in-memory index tables and append-only value logs. We will present some numbers on how GriffinDB performs on both on-premise and AWS environments.
Software versions
Linux OS: Ubuntu 18.04 Desktop and CentOS 7.5
Metadium: October 2018 and later versions
Questions?
Contact info@circuitblvd.com for additional information.
Comments