First published on VOICE in October 2020 -> Check it out here.
While we can’t yet dive inside the super-secret block.one GitHub repositories, we can view the ongoing public development and envisage future capabilities and Use cases.
In this eosio developments series #2, we look into some of the potential changes coming to the codebase, in particular software related to the storage of transactional data.
Current State
In its current form, nodeos (the core EOSIO node daemon) stores blockchain data in what’s known as RAM. Those familiar with personal computer architecture will know RAM is the superfast, temporary or volatile, Random Access Memory which is directly available to the computer’s CPU. There is more to it, but for our comparison, the definition is adequate.
Eosio employs the notion of RAM, minus the volatile or temporary aspect. The chain software stores Smart Contract data in what is known as multi-index tables, inside an in-memory database stored in the physical server’s memory.
While utilising physical RAM is fantastic for application performance and was a splendid way to get the EOS public blockchain bootstrapped, like anything in technology, it comes with trade-offs.
Let’s say you’re like me and have 1000s of family photos. Storing your entire collection in your computer’s memory is not practical as you quickly run out available space. Instead, you move your infrequently accessed files to an external hard drive, network-attached storage or a cloud service like Google Photos to keep things efficient and cost-effective. Storing photos in memory would be excessive for most; however, say you did have unlimited resources and a home-sized supercomputer it would provide you with the ultimate in personal photo editing performance and capability.
As stated earlier, a similar trade-off exists within the eosio software. On launch, eosio shipped with a single tier of storage called RAM. Storage in a decentralised Blockchain network is a scarce resource. Every node in the system must offer the same capacity to enable data to be replicated, making it accessible to the users of the network. Furthermore, storage (EOS specifically) is becoming a considerable burden for node operators. Building servers with vast quantities of physical RAM is not practical for long term network health.
This trade-off resulted in solutions like vRAM from LiquidApps, which strang to life soon after the EOS Mainnet launch. The concept offers developers an alternative, semi on-chain tier of storage which frees up space, reduces demand and limits operational data stored on-chain.
Future State
But wait, Hard Disks are cheap! Why is this such a problem? Remember RAM on EOS and eosio lives in RAM, and indirectly on Hard Drives.
The below chart (source: hblok.net) shows the historical price differences between high-speed storage vs less performant options. You can see the difference between DIMMs/memory (green) and the others.
For eosio to be a successful blockchain and database alternative, enabling scale for mass-market adoption necessitates a lower-cost storage alternative. Notice I said, lower-cost not lower performance? If you do some digging, you’ll see there are new technologies which dramatically improve read/write performance of storage at a lower price point.
If you’re interested, check out Intel Optane or 3D XPoint Technology.
So what’s B1 doing to improve the situation?
Enter RocksDB, a key/value database created by FaceBook, which is open source and usable by anyone.
The RocksDB library provides a persistent key-value store. Keys and values are arbitrary byte arrays….The library is maintained by the Facebook Database Engineering Team and is based on LevelDB.
RocksDB is known for powering some of the world’s most popular platforms such as FaceBook and Netflix. It provides a vast array of tuning options for many different hardware types.
How does this tie into eosio?
This pull request, introduces the concept of using RocksDB or a Key-Value API within eosio software. In this fantastic article by LiquidApps, the team breaks down a new type of storage coming to eosio. At the time it was called DISK and looked to be an extension of our computer analogy.
However, in recent times, Block.One appears to have walked away from the notion of an independent DISK tier. This PR removes the concept of Disk, replacing it with what’s called a backing-store.
The backing-store, has two options. 1) Chainbase, and 2) RocksDB. For those not familiar with the Chainbase, the full definition describes it as a transactional database used for Blockchains. Chainbase gets used in eosio and EOS today and is the basis for RAM.
What’s interesting is option 2. RocksDB now appears to be a full replacement for Chainbase, rather than a complementary storage layer. My guess during testing, RocksDB proved it was capable enough to shoulder full responsibility for RAM and hence DISK wasn’t required.
In my view, this is a great outcome for anyone building chains or applications on eosio. Assuming I’m on the right track, and this doesn’t change dramatically, rather than having to manage the complexity of multiple storage layers, a contract can interact with RAM and the KVAPI directly. It means eosio is analogous to traditional databases where the complexity of storage subsystems are abstracted or handled by the storage hardware.
Any time you can reduce complexity, you improve the chances of general takeup. People converge around the most straightforward solutions, so keeping storage fast and direct should help to enhance eosio usability.
Final note, RocksDB can scale! Using Rocks as a storage engine opens the door for high capacity, cost-effective, fast, decentralised storage on any eosio based network. With this technology, you’ll be able to keep more data on-chain and available! Exciting times for infrastructure, blockchains and decentralised applications.
Stay tuned for future development series updates.
