# Simulate Production in Docker
Make sure you have everything you need before proceeding:
- You understand the concepts of production.
- Docker is installed.
- You have the checkers blockchain codebase with the CosmJS elements. If not, follow the previous steps or check out the relevant version (opens new window).
In this section, you will:
- Prepare Docker elements.
- Handle keys.
- Prepare blockchain nodes.
- Prepare a blockchain genesis.
- Compose the lot in one orchestrated ensemble.
Before you launch yourself fully into production, it would be interesting to simulate a set of independent nodes on your computer. This can be prepared and orchestrated with Docker Compose (opens new window).
# Target setup
In order to mimic a real setup, find an objective on which to focus. Here the objective is:
- Three independent parties - Alice, Bob, and Carol.
- Two independent validator nodes, run by Alice and Bob respectively, that can only communicate with their own sentries and do not expose RPC endpoints.
- Additionally, Alice's validator node uses Tendermint Key Management System (TMKMS) on a separate machine.
- The two sentry nodes, run by Alice and Bob, expose endpoints to the world.
- A regular node, run by Carol, that can communicate with only the sentries and exposes endpoints for use by clients.
# Docker elements
Before looking at the specific Compose elements, you need to define what the regular Docker elements are.
You will run containers. You can start by giving them meaningful names:
- Alice's containers:
- Bob's containers:
- Carol's containers:
Docker lets you simulate private networks. To meaningfully achieve the above target setup in terms of network separation, you use Docker's user-defined networks. This means:
- Alice's validator and key management system (KMS) are on their private network: name it
- Alice's validator and sentry are on their private network: name it
- Bob's validator and sentry are on their private network: name it
- There is a public network on which both sentries and Carol's node run: name it
Although every machine on the network is a bit different, in terms of Docker images there are only two image types:
- The Tendermint nodes (validators, sentries, and regular nodes) will run
checkersdwithin containers created from a single Docker image.
- The Tendermint KMS node will run TMKMS from a different Docker image.
# The node image
The node image contains, and runs by default, the checkers executable. You first have to compile it, and then build the image.
First, build the executable(s) that will be launched by Docker Compose within the Docker images. Depending on your platform, you will use
If you have a CPU architecture that is neither
arm64, update your
If you copy-pasted directly into
Makefile, do not forget to convert the spaces into tabs.
Now run either command:
Now include the relevant executable inside your production image. Create a new
Build the image with:
Depending on your installed version of Docker, you may have to add the flags:
Or just manually replace
amd64 or whichever is your architecture.
Now you can run it:
You should see a recognizable list of commands.
Each Docker container will run
root, which does not matter because it all happens in a container. Therefore, there is no need to create a specific additional user like you would in a serious production setting. For the same reason, there is also no need to create a service to launch it.
# The key manager image
Alice runs the Tendermint Key Management System (opens new window) on a separate machine. You need to prepare its image. The image will contain the executable, which you have to compile from its Rust code.
There are several considerations to clarify:
- How will you build it?
- What device will store the key?
- What KMS version works with your node version?
The build step is a good opportunity to use a multi-stage Docker build (opens new window). With this technique:
- You define a disposable image (the first stage) that clones the code and compiles it. This involves the download of Rust crates (i.e. packages). This image ends up being large but is then disposed of.
- You define a slim image (the second stage) in which you only copy the compiled file. This is the image you keep for production. It ends up being small.
The disposable image needs to use Rust of at least version 1.56. Fortunately, there are ready-made Docker images. Pick
rust:1.64.0 (opens new window).
Next, the executable needs to be compiled for the specific device onto which your key will be stored. You do not use hardware keys in this setup. So, when building it, you use the
softsign extension (opens new window). This is achieved by adding the flag
Finally, what version of the TMKMS should you compile? A given TMKMS version can work with a limited set of specific Tendermint versions. Find the Tendermint version of your checkers code with:
It should return something like this:
Because here it is version 0.34, it is a good idea to use the KMS from version 0.10.0 (opens new window) upwards. At the time of writing, version 0.12.2 still seems to support Tendermint v0.34. It is under the
v0.12.2 (opens new window) tag on Github. Pick this one.
Having collected the requisites, you can create the multi-staged Docker image in a new
As you can see, the production stage is only three lines. Build the image as usual with:
Now run it:
It returns you information about usage. You have just built the Tendermint Key Management System.
# Blockchain elements
Each container needs access to its private information, such as keys, genesis, and database. To facilitate data access and separation between containers, create folders that will map as a volume to the default
/root/tmkms inside containers. One for each container:
For instance, when running a container for
val-alice, you would create the volume mapping with a command like:
And for the KMS, like so:
# Basic initialization
Before you can change the configuration you need to initialize it. Do it on all nodes with this one-liner:
As a secondary effect, this also creates the first shot of
config/genesis.json on every node, although you will start work with the one on
Early decisions that you can make at this stage are:
- Deciding that the chain will be named
checkers-1. It is a convention to append a number in case it has to go through a hard fork.
- Deciding that the staking denomination will be called
upawn, which will be understood as 1 PAWN equals 1 million of
Do you need that many decimals? Yes and no. Depending on your version of the Cosmos SDK, there is a hard-coded value of base tokens that a validator has to stake, and the number is
10,000,000. If you do not have enough decimals, the human token would have to have a lot of zeroes.
The default initialization sets the base token to
stake, so to get it to be
upawn you need to make some changes:
- In the authoritative
config/genesis.json(opens new window) (
Note how the command overrides the default
checkersd entry point and replaces it with
- In all five
config/app.toml(opens new window):
Make sure that
config/client.toml (opens new window) mentions
checkers-1, the chain's name:
Some keys are created automatically, like the node keys (opens new window). For others, you have to create them yourself. You will create:
- The validator operator keys for Alice and Bob.
- The consensus keys, whether they stay on Bob's node or are kept inside Alice's KMS.
Start with the keys for the validators and Alice's KMS Tendermint key.
# Validator operator keys
First, you need to create the two validators' operation keys. Such a key is not meant to stay on the node when it runs, it is meant to be used at certain junctures only, for instance, to stake on behalf of Alice (or Bob). Nonetheless, you are going to create them by running containers. Because you want to keep these keys inside and outside of containers, do the following:
- Use the
- Keep them in the mapped volume with
Create the operator key for
Use a passphrase you can remember. It does not need to be exceptionally complex as this is all a local simulation. This exercise uses
password and stores this detail on file, which will become handy.
Because with this prod simulation you care less about safety, so much less in fact, you can even keep the mnemonic on file too.
Do the same for
# Alice's consensus key on the KMS
To get the KMS to work, you have to:
- Prepare the KMS.
- Import Alice's consensus key into the KMS'
- Have the KMS and the node talk to each other.
# Prepare the KMS
As per the documentation (opens new window), initialize the KMS folder:
In the newly-created
Make sure that you use the right protocol version. In your case:
Pick an expressive name for the file that will contain the
checkers-1, the name of your blockchain, wherever the former appears:
# Import the consensus key
Now you need to import
val-alice's consensus key in
The private key will no longer be needed on
val-alice. However, during the genesis creation Alice will need access to her consensus public key. Save it in a new
pub_validator_key.json (opens new window) without any new line:
The consensus private key should not reside on the validator. You can simulate that by moving it out:
Import it into the
softsign "device" as defined in
tmkms.toml (opens new window):
val-alice may still recreate a missing private key file due to how defaults are handled in the code. To prevent that, you can instead copy it from
sentry-alice where it has no value.
With the key created you now set up the connection from
# Set up the KMS connection
Choose a port unused on
val-alice, for instance
26659, and inform
In the above,
val-alice is the future network name of Alice's validator, and it will indeed be resolved to an IP address via Docker's internal DNS. In a real production setup, you would use a fully resolved IP address to avoid the vagaries of DNS.
Do not forget, you must inform Alice's validator that it should indeed listen on port
Make it listen on an IP address that is within the KMS private network.
0.0.0.0 represents all addresses of the node. In a real production setup, you would choose the IP address of the network card that is on the network common with
- Make sure it will not look for the consensus key on file:
- Make sure it will not look for the consensus state file either, as this is taken care of by the KMS:
Before moving on, make sure that the validator still has a
priv_validator_key.json because the code may complain if the file cannot be found. You can copy the key from
sentry-alice, which does not present any risk:
With the keys in you can start fleshing out the genesis, which is already created.
You need to:
- Set up the chain ID.
- Add the initial balances.
- Add the initial validator stakes.
- Distribute the genesis file to all relevant nodes.
# Set up chain ID
Earlier you chose
checkers-1, so you adjust it here too:
# Initial balances
In this setup, Alice starts with 1,000 PAWN and Bob with 500 PAWN, of which Alice stakes 60 and Bob 40. With these amounts, the network cannot start if either of them is offline. Get their respective addresses:
password with the passphrase you picked when creating the keys.
Have Alice add her initial balance in the genesis:
Now move the genesis file to
val-bob. This mimics what would happen in a real-life setup:
Have Bob add his own initial balance:
# Initial stakes
Alice and Bob both have initial stakes that they define via genesis transactions. You create them.
# Bob's stake
Bob is not using the Tendermint KMS but instead uses the validator key on file. Bob appears in second position in
app_state.accounts, so his
account_number ought to be
1, but it is in fact written as
0, so you use
Again, insert Bob's chosen passphrase instead of
password. Return the genesis to Alice:
It is Alice's turn to add her staking transaction.
# Alice's stake
Create Alice's genesis transaction using the specific validator public key that you saved on file, and not the key that would be taken from
priv_validator_key.json by default (and is now missing):
It is useful to know this
--pubkey method. If you were using a hardware key located on the KMS, this would be the canonical way of generating your genesis transaction.
# Genesis assembly
With the two initial staking transactions created, have Alice include both of them in the genesis:
As an added precaution, confirm that it is a valid genesis:
It should return:
# Genesis distribution
All the nodes that will run the executable need the final version of the genesis. Copy it across:
# Network preparation
Because the validators are on a private network and fronted by sentries, you need to set up the configuration of each node so they can find each other; also to make sure that the sentries keep the validators' addresses private. What are the nodes' public keys? For instance, for
val-alice, it is:
This returns something like:
# Set up Alice's sentry
The nodes that have access to
val-alice should know Alice's sentry by this identifier:
val-alicewill be resolved via Docker's DNS.
26656is the port as found in
In the case of
sentry-alice has access to it. Moreover, this is a persistent node. So you add it in
sentry-alice also has access to
node-carol, although these nodes should probably not be considered persistent. You will add them under
"seeds". First, collect the same information from these nodes:
sentry-alice, you should have:
Before moving on to other nodes, remember that
sentry-alice should keep
val-alice secret. Set:
# Other nodes
Repeat the procedure for the other nodes, taking into account their specific circumstances:
For the avoidance of doubt,
sentry-alice has a different address depending on which node resolves the address:
- When it is resolved from
val-alice, the resolution takes place in
- When it is resolved from
sentry-bob, the resolution takes place in
# Open Carol's node
Carol created her node to open it to the public. Make sure that her node's RPC listens on all IP addresses:
# Compose elements
You have prepared:
- The basic Docker elements
- The blockchain elements
- The network elements
Time to assemble them in Compose.
# The executables that run
You define the different containers as
services. Important elements to start with are:
container_name, you use names that make them intelligible and match the names you used in the above preparations.
image, you declare the Docker image to use.
command, you define the command to use when launching the image.
In a new
docker-compose.yml (opens new window), write:
You are going to further refine the service definitions next, starting with the disk volumes.
# The data each container needs
Each container needs to access its own private folder, prepared earlier, and only that folder. Declare the volume mappings:
# The networks they run in
The user-defined networks need to mimic the desired separation of machines/containers can be self-explanatorily declared as:
With the network declaration done, the associating of each computer to each network can be written as:
# Additional settings
The KMS connects to the node and can reconnect. So have
val-alice start after
With all these computers on their Docker networks, you may still want to access one of them to query the blockchain, or to play games. In order to make your host computer look like an open node, expose Carol's node on all addresses of your host:
# Launching Compose
After this long preparation, before launch, it could be a good time to make a Git commit so that you can restore easily.
You are now ready to start your setup with a name other than the folder it is running in:
At this point, it should be apparent that you need to update
priv_validator_state.json is necessary if you want to try again on another host, otherwise, it would be ignored by Git.
# Interacting with Compose
Your six containers are running. To monitor their status, and confirm that they are running, use the provided Docker container interface.
Now you can connect to
node-carol to start interacting with the blockchain as you would a normal node. For instance, to ask a simple
From this point on everything you already know how to do, such as connecting to your local node, applies.
Whenever you submit a transaction to
node-carol, it will be propagated to the sentries and onward to the validators.
At this juncture, you may ask: Is it still possible to run a full game in almost a single block as you did earlier in the CosmJS integration tests? After all, when
node-carol passes on the transactions as they come, it is not certain that the recipients will honor the order in which they were received. Of course, they make sure to order Alice's transactions, thanks to the
sequence, as well as Bob's. But do they keep the A-B-A-B...order in which they were sent?
To find out:
- You connect within
RPC_URL="http://node-carol:26657"(opens new window).
- You use the Alice and Bob of this document as the CosmJS test Alice and Bob. Copy the content of
MNEMONIC_TEST_ALICE(opens new window). Then do the same for Bob.
- Update the
ADDRESS_TEST_ALICE(opens new window) address to contain the correct ones, and repeat the same for Bob.
- You connect within
Skip the call to the (Ignite) faucet by adding a
returnin the relevant
beforein the test file:
Double all the timeouts. For instance:
Text-replace all the
token(opens new window) with
Launch the lot within
Tests should pass. Should as in there is no protocol guarantee that they will, but it looks like they do.
# Stopping Compose
To stop your whole setup, run:
You may encounter an error such as:
If encountering this or other errors, you may want to do a state reset on all nodes:
If one of your services (for example,
sentry-bob) fails to start because it could not resolve one of the other containers, you can restart that service independently with:
If you want to get more detailed errors from your KMS, you can add a flag in its service definition:
If you want to erase all states after a good run, and if you have a Git commit from which to restore the state files, you can create a new script (opens new window) for that.
To summarize, this section has explored:
- How to prepare Docker images.
- How to prepare nodes for a simulated production setup.
- How to prepare a Tendermint Key Management System for a simulated production setup.
- How to prepare a blockchain genesis with multiple parties.
- How to launch all that with the help of Docker Compose.