5. Fun With The Bitcoin Blockchain

For this tutorial we will use the leveldb_blockchain backend, which uses the LevelDB database to store the Bitcoin blockchain. Backends implement the blockchain interface. Starting and stopping a backend is down to the individual implementation.

The blockchain can consume a large number of open file handles during normal operation. In particular, the LevelDB backend may accumulate a number of data files. The creation of numerous data files is normal. Most operating systems can change the open-files limit using the ulimit -n command. Example:

ulimit -n 4096

However, this only changes the limit for the current shell session. Changing the limit on a system-wide, permanent basis varies more between systems. See this guide for more information on open files limits.

5.1. Initialization: Importing The Genesis Block

Before we can use leveldb_blockchain, the database needs to be initialised.

  1. Create a leveldb_blockchain object. If no database exists in the specified path, it will automatically be created.
  2. Recreate the genesis block. The first block in the Bitcoin blockchain is part of the specification for the Bitcoin standard.
  3. Import the genesis block at height 0 in the blockchain.

You now have a working leveldb_blockchain.

To create the database which initialises a new one if one doesn’t already exist, we call leveldb_blockchain::start() with a root path. The completion handler is called passing an std::error_code that indicates whether the operation was successful or not.

// Threadpool context containing 1 thread.
threadpool pool(1);
// leveldb_blockchain operations execute in pool's thread.
leveldb_blockchain chain(pool);
// Completion handler for starting the leveldb_blockchain.
// Does nothing.
auto blockchain_start = [](const std::error_code& ec) {};
// Start blockchain with a database path.
chain.start(dbpath, blockchain_start);

The function genesis_block() returns a genesis block.

// First block is the genesis block.
block_type first_block = genesis_block();

We call blockchain::import() to save a block in the blockchain at a specified height directly. It doesn’t validate or perform any safety checks on the imported block. Instead the block is written directly.

// Completion handler for import method.
auto import_finished =
    [&ec_promise](const std::error_code& ec)
    {
        ec_promise.set_value(ec);
    };
// Import the genesis block at height 0.
// Doesn't validate or perform checks on the block.
chain.import(first_block, 0, import_finished);
// Wait until std::error_code is set by
// import_finished completion handler.
std::error_code ec = ec_promise.get_future().get();
if (ec)
{
    log_error() << "Importing genesis block failed: " << ec.message();
    return -1;
}

All operations need to finish on leveldb_blockchain before it can be closed properly so we first stop the threadpool before calling leveldb_blockchain::stop().

// All threadpools stopping in parallel...
pool.stop();
// ... Make them all join main thread and wait until they finish.
pool.join();
// Now safely close leveldb_blockchain.
chain.stop();

blockchain::store() is the recommended way to add new blocks to the blockchain. It finds the correct height by looking up the previous block, handles reorganisations, validates the blocks and calls the subscription handlers.

void blockchain::store(const block_type& block, store_block_handler handle_store)

Store a new block.

Subscriber is notified exactly once of changes to the blockchain and needs to re-subscribe to continue being notified.

void handle_store(
    const std::error_code& ec,   // Status of operation
    block_info info              // Status and height of block
);

The full sourcecode can be found in examples/initchain.cpp.

5.2. Fetch and Display Block Info

Services like blockchain do not block. Methods return immediately and upon completion call a completion handler. The semantics of the blockchain reflect this with the set/get_* methods being equivalently called store/fetch_*.

The only thing we add to the blockchain is new blocks. There is one method called blockchain::store(). This method handles the internal details of validating the block against the current blockchain, returning competing blocks to the orphan pool (if needed), insertion into the database and processing any dependent blocks.

In our example we want to fetch and display the last block header. To fetch the last height number, we use blockchain::fetch_last_height(). To fetch the block header for a height number, we use blockchain::fetch_block_header().

void blockchain::fetch_block_header(size_t height, fetch_handler_block_header handle_fetch)

Fetches the block header by height.

void handle_fetch(
    const std::error_code& ec,      // Status of operation
    const block_header_type& blk    // Block header
);
void blockchain::fetch_last_height(fetch_handler_last_height handle_fetch)

Fetches the height of the last block in our blockchain.

void handle_fetch(
    const std::error_code& ec, // Status of operation
    size_t block_height        // Height of last block
);

All the blockchain fetch methods give you access to all of the data in the blockchain to reconstruct or link any piece of data. Full navigation around the chain is possible.

Starting at the basic level, we start with an application to start the blockchain otherwise report the error back.

We create a threadpool, blockchain, and then call start. Then after when the user is ready to exit, we stop and join the threadpool, and safely close the blockchain.

#include <bitcoin/bitcoin.hpp>
using namespace bc;

blockchain* chain = nullptr;

// Completion handler for when the blockchain has finished initializing.
void blockchain_started(const std::error_code& ec)
{
    // std::error_code's can be tested like bools, and
    // compared against specific error enums.
    // See <bitcoin/error.hpp> for a full list of them.
    if (ec)
    {
        log_error() << "Blockchain failed to start: " << ec.message();
        return;
    }
    // Blockchain has safely started.
    log_info() << "Blockchain started.";
}

int main()
{
    // Define a threadpool with 1 thread.
    threadpool pool(1);
    // Create a LevelDB blockchain.
    leveldb_blockchain ldb_chain(pool);
    // Initialize our global 'chain' pointer from above.
    chain = &ldb_chain;
    // Start the database using its implementation specific method.
    ldb_chain.start("../database", blockchain_started);
    // Keep running until the user presses enter.
    // Since libbitcoin is asynchronous, you need to synchronise with
    // them to know when to exit safely.
    // For these examples we just pause until enter for simplicity sake.
    std::cin.get();
    // Begin stopping the threadpools in parallel (only 1 here).
    pool.stop();
    // Join them one by one.
    pool.join();
    // Finally stop the blockchain safely now everything has stopped.
    ldb_chain.stop();
    return 0;
}

After the blockchain has started, we want to begin the entire process. The process starts with getting the last height in our blockchain, then fetching the block header at that height, and finally displaying the block header to the screen.

// Completion handler for when the blockchain has finished initializing.
void blockchain_started(const std::error_code& ec);
// Fetch the last block now that we have the height.
void height_fetched(const std::error_code& ec, size_t last_height);
// Result: print the block header.
void display_block_header(const std::error_code& ec,
    const block_header_type& header);

After the blockchain has started, we begin the operation to fetch the last height, calling height_fetched() after it’s finished.

void blockchain_started(const std::error_code& ec)
{
    // std::error_code's can be tested like bools, and
    // compared against specific error enums.
    // See <bitcoin/error.hpp> for a full list of them.
    if (ec)
    {
        log_error() << "Blockchain failed to start: " << ec.message();
        return;
    }
    // Blockchain has safely started.
    log_info() << "Blockchain started.";
    // chain should've been set inside main().
    assert(chain);
    // Begin fetching the last height number.
    chain->fetch_last_height(height_fetched);
}

After height_fetched() has been called, we know the block number and begin fetching the block header.

void height_fetched(const std::error_code& ec, size_t last_height)
{
    if (ec)
    {
        log_error() << "Failed to fetch last height: " << ec.message();
        return;
    }
    // Display the block number.
    log_info() << "Height: " << last_height;
    assert(chain);
    // Begin fetching the block header.
    chain->fetch_block_header(last_height, display_block_header);
}

Now finally the block header is received, and can be displayed. This is the final operation in this sequence.

As we only requested the block header, the transactions list will be empty. Getting a full block involves getting the transaction hashes associated with that block, and fetching each one which is provided by the composed operation fetch_block().

void display_block_header(const std::error_code& ec,
    const block_header_type& header)
{
    if (ec)
    {
        log_error() << "Failure fetching block header: " << ec.message();
        return;
    }
    // 32 byte std::array of uint8_t
    const hash_digest& blk_hash = hash_block_header(header);
    // Encode block hash into a pretty hex string.
    log_info() << "hash: " << encode_hex(blk_hash);
    // Display a few fields from the block header.
    // See <bitcoin/primitives.hpp> for the definition of block_type.
    log_info() << "version: " << header.version;
    // hash_digest can be used directly in log_info(),
    // implicity calling encode_hex() on the hash_digest.
    log_info() << "previous_block_hash: " << header.previous_block_hash;
    log_info() << "merkle: " << header.merkle;
    log_info() << "timestamp: " << header.timestamp;
    log_info() << "bits: " << header.bits;
    log_info() << "nonce: " << header.nonce;
    // A goodbye message.
    log_info() << "Finished.";
}

The full example is in examples/display-last.cpp.

5.3. Message from Satoshi, Bitcoin’s creator

Satoshi left us a message inside the first Bitcoin genesis block.

// The Times 03/Jan/2009 Chancellor on brink of second bailout for banks

The message is inside the first input, of the first transaction, of the first Bitcoin block.

Block 0 is predefined by Bitcoin. All blockchains must begin with the same block otherwise they aren’t Bitcoin. genesis_block() recreates a copy of block 0.

  1. Create genesis block.
  2. Lookup first transaction in block (the coinbase transaction).
  3. Get the first input from the coinbase transaction.
  4. Serialize the input’s script back into raw form.
  5. Display the raw input script.

The input script for the first input of the coinbase transaction inside the genesis block contains the message from Satoshi.

// examples/satoshiwords.cpp
#include <bitcoin/bitcoin.hpp>
using namespace bc;

int main()
{
    // Create genesis block.
    block_type genblk = genesis_block();
    // Genesis block contains a single coinbase transaction.
    assert(genblk.transactions.size() == 1);
    // Get first transaction in block (coinbase).
    const transaction_type& coinbase_tx = genblk.transactions[0];
    // Coinbase tx has a single input.
    assert(coinbase_tx.inputs.size() == 1);
    const transaction_input_type& coinbase_input = coinbase_tx.inputs[0];
    // Get the input script (sometimes called scriptSig).
    const script& input_script = coinbase_input.input_script;
    // Convert this to its raw format.
    const data_chunk& raw_block_message = save_script(input_script);
    // Convert this to an std::string.
    std::string message;
    message.resize(raw_block_message.size());
    std::copy(raw_block_message.begin(), raw_block_message.end(),
        message.begin());
    // Display the genesis block message.
    std::cout << message << std::endl;
    return 0;
}

5.4. Reconstruct Block Transactions

To reconstruct an entire block from a block header, first obtain a list of transaction hashes that makeup that block. Then iterate the list of transaction hashes, fetching the transactions one by one.

void blockchain::fetch_block_transaction_hashes(const hash_digest& block_hash, fetch_handler_block_transaction_hashes handle_fetch)

Fetches list of transaction hashes in a block given the block hash.

void handle_fetch(
    const std::error_code& ec,      // Status of operation
    const inventory_list& hashes    // List of hashes
);
void blockchain::fetch_transaction(const hash_digest& transaction_hash, fetch_handler_transaction handle_fetch)

Fetches a transaction by hash

void handle_fetch(
    const std::error_code& ec,  // Status of operation
    const transaction_type& tx  // Transaction
);

5.4.1. fetch_block() and Composed Operations

libbitcoin provides a convenience function fetch_block() to wrap the details of fetching a full block. These kind of operations that wrap a bunch of other operations are called composed operations.

A general design principle of libbitcoin is to keep the implementation simple and not pollute class interfaces. Instead composed operations wrap lower level class methods to simplify common operations.

void fetch_block(blockchain& chain, size_t height, blockchain_fetch_handler_block handle_fetch)

Fetch a block by height. If the blockchain reorganises, operation may fail halfway.

void handle_fetch(
    const std::error_code& ec,  // Status of operation
    const block_type& blk       // Block header
);

5.5. Polling Blocks From Nodes

The poller service downloads blocks from nodes into the blockchain.

// ...

void connection_established(const std::error_code& ec, channel_ptr node,
    poller& poll)
{
    // ...
    // getblocks request asking node for a list of blocks to download.
    // Usually you call query() on the first node you connect to.
    poll.query(node);
    // Monitor for inventory packets containing blocks we don't have.
    // Then request and attempt to store the blocks in the blockchain.
    poll.monitor(node);
}

int main()
{
    threadpool pool(2);
    leveldb_blockchain chain(pool);
    // ...
    poller poll(pool, chain);
    // ...
    return 0;
}

poller handles the details of watching for notification of new blocks, sending requests as needed and storing them in the blockchain by calling blockchain::store().

5.5.1. Reorganizations And New Blocks

While polling new blocks from the network, callbacks registered with blockchain::subscribe_reorganize() will be notified of any changes to the blockchain.

void blockchain::subscribe_reorganize(reorganize_handler handle_reorganize)

Be notified of the next blockchain change.

Subscriber is notified exactly once of changes to the blockchain and needs to re-subscribe to continue being notified.

void handle_reorganize(
    const std::error_code& ec,   // Status of operation
    size_t fork_point,           // Index where blockchain forks
    const block_list& added,     // New blocks added to blockchain
    const block_list& removed    // Blocks removed (empty if none)
);

The fork_point gives the height of the ancestor block before the split. Both lists are ordered from lowest height first.

for (size_t i = 0; i < added_blocks.size(); ++i)
{
    size_t height = fork_point + 1 + i;
    const block_type& blk = *added_blocks[i];
}

5.6. Optimization Techniques

  • Create a separate partition for the database directory with the noatime parameter set.
  • Lower VM swappiness.
  • Increase the max number of open files.
  • Disable filesystem caching to prevent double caching by database and operating system.
  • Make heavy use of replication i.e multiple blockchain instances each with their own database.
  • Follow the guidelines here.