Skip to content

Commit

Permalink
Review and edit Narwhal Docker README (MystenLabs#299)
Browse files Browse the repository at this point in the history 
* Update README.md

* Update README.md

* Update Docker/README.md

Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>

* Update Docker/README.md

Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>

Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>
  • Loading branch information
@Clay-Mysten @huitseeker
Clay-Mysten and huitseeker authored Jun 7, 2022
1 parent 6043a7c commit c141393
Showing 1 changed file with 94 additions and 89 deletions.
183 changes: 94 additions & 89 deletions narwhal/Docker/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,107 +1,111 @@
### Introduction
# Narwhal Cluster Startup

This directory contains all the necessary configuration to allow someone
quickly setup and spin-up a small Narwhal cluster via [docker-compose](https://docs.docker.com/compose/).
## Introduction

Under this directory there will be found the following 2 things:
* The `Docker` file definition for a Narwhal node
* A `docker-compose` file to allow someone quickly spin-up a Narwhal cluster
This directory contains the configuration information needed to
quickly setup and spin-up a small Narwhal cluster via [Docker Compose](https://docs.docker.com/compose/).

### Quick start
In this directory, you will find:
* The `Dockerfile` definition for a Narwhal node
* A `docker-compose.yml` file to allow you to quickly create a Narwhal cluster

The following dependencies must be installed before continuing further.
## Quick start

* **Docker:** Please find installation info [here](https://docs.docker.com/get-docker/)
* **Docker-compose:** Please find installation info [here](https://docs.docker.com/compose/install/)
First, you must install:

After having installed `Docker` & `docker-compose`, next step will be to
start the cluster.
* [Docker](https://docs.docker.com/get-docker/)
* [Docker-compose](https://docs.docker.com/compose/install/)

Before everything **make sure that you are on the `Docker folder`** . On the rest of the
document we'll assume that we are under this folder.
Afterward, you will start the Narwhal cluster.

First, **make sure that you are on the `Docker folder`** . In the rest of the
document, we'll assume that we are under this folder:
```
$ cd Docker # Change to Docker directory
$ pwd # Print the current directory
narwhal/Docker
```

then bring up the cluster via the following command:
Then bring up the cluster via the following command:
```
$ docker-compose -f docker-compose.yml up
```
The first time this will run, it will build the narwhal docker image (this can take a few minutes
since the narwhal node binary needs to be built from the source code) and then it will spin up
a cluster for `4 nodes` by doing the necessary setup for `primary` and `worker` nodes. Each
`primary` node will be connected to `1 worker` node.

The logs for each validator (primary & worker nodes) can be found on the logs folder under the corresponding
The first time this runs, `docker-compose` will build the Narwhal docker image. (This can take a few minutes
since the narwhal node binary needs to be built from the source code.) And then it will spin up
a cluster for *four nodes* by doing the necessary setup for `primary` and `worker` nodes. Each
`primary` node will be connected to *one worker* node.

The logs for each validator (primary & worker nodes) can be found in the `logs` folder under the corresponding
validator folder.
_______
**Validator:** You will find across the codebase the name `Authority` when we basically want to describe
the entity that is participating on the Narwhal protocol that verifies and signs the produced headers. An
authority is consisted of a `primary` node and one or more `worker` nodes. Since `Authority` is not a widely
used term, we are referring instead to our `primary + worker` node pairs as `Validator`.
_______

The `logs` folder will be created once the node is bootstrapped via docker-compose.
For example, for the primary node of the validator-0, the logs will be found under
the folder [logs](validators/validator-0/logs) and with the name `log-primary.txt`. To monitor the logging
of a node in real time you can just do something like:

> **Note**: You will find across the codebase the name *Authority* when we describe
> the entity participating in the Narwhal protocol that verifies and signs the produced headers. An
> authority is composed of a `primary` node and one or more `worker` nodes. Since Authority is not a widely
> used term, we are referring instead to our *primary + worker* node pairs as **Validator**.
The `logs` folder is created once the node is bootstrapped via `docker-compose`.
For example, for the primary node of the validator-0, the logs will be found in
the folder `validators/validator-0/logs` with the name `log-primary.txt`. To monitor
the logging of a node in real time, simply `tail` the log:
```
$ tail -f validators/validator-0/logs/log-primary.txt
```

By default, the production (release) version of the Narwhal node will be compiled when the Docker image is being built.
To build the Docker image with the development version of it, which will lead to smaller compile times and
smaller binary (and image) size, you can run the docker-compose command as:
To build the Docker image with the development version of Narwhal, which will lead to smaller compile times and
smaller binary (and image) size, you run the `docker-compose` command as:
```
$ docker-compose build --build-arg BUILD_MODE=debug
```

# and then run as
And then run:

```
$ docker-compose up
```

**Warning**: by default each validator's directory will be cleaned up between docker-compose runs when each node
bootstraps. To preserve those between runs please see the usage of the environment variable `CLEANUP_DISABLED` on
the [section](#docker-compose-configuration) bellow.
> **Warning**: By default, each validator's directory will be cleaned up between `docker-compose` runs when each node
> bootstraps. To preserve those logs between runs, employ the environment variable `CLEANUP_DISABLED` as described in
> [Docker Compose configuration](#docker-compose-configuration).
### Build Docker image without docker-compose
## Build Docker image without docker-compose

To build the Narwhal node image without docker-compose the following command should be used:
To build the Narwhal node image without `docker-compose`, run:
```
$ docker build -f Dockerfile ../ --tag narwhal-node:latest
```

Since the [Dockerfile](Dockerfile) is located under a different folder other than the source code,
it is important to define the context `../` and allow the Dockerfile properly COPY the source
code to be compiled on later steps.
Since the [Dockerfile](Dockerfile) is located under a different folder from the source code,
it is important to define the context (ex. `../`) and allow the Dockerfile to properly **copy**
the source code to be compiled in later steps.

### Access primary node public gRPC endpoints
## Access primary node public gRPC endpoints

The nodes by default are running with the `Tusk` algorithm disabled, which basically allow
to user to treat Narwhal as a pure mempool. When that happens, the gRPC server is bootstrapped
for the primary nodes and that allow someone to interact with the node (ex the consensus layer).
The nodes by default are running with the `Tusk` algorithm disabled, which basically allows
you to treat Narwhal as a pure mempool. When that happens, the gRPC server is bootstrapped
for the primary nodes, and that allows interaction with the node (ex. the consensus layer).

The gRPC server for a primary node is running on port `8000`. However, by default, a container's port
is not accessible to hit by the host (local) machine unless it's exported a mapped between a host's
machine port and the corresponding container's port (ex for someone to use a gRPC client on their
computer to hit a primary's node container gRPC server). The [docker-compose](docker-compose.yml) file is
exporting the gRPC port for each primary node, so they can be accessible from the host machine.
For the default setup of `4 primary` nodes, the gRPC servers are basically listening to the following
is not accessible to hit by the host (local) machine unless it's exported a mapping between a host's
machine port and the corresponding container's port (ex. for someone to use a gRPC client on their
computer to hit a primary's node container gRPC server). The [docker-compose.yml](docker-compose.yml) file
exports the gRPC port for each primary node so they can be accessible from the host machine.

For the default setup of *four primary* nodes, the gRPC servers are listening to the following
local (machine) ports:
* `primary_0`: 8000
* `primary_1`: 8001
* `primary_2`: 8002
* `primary_3`: 8003

For example, to send a gRPC request to `primary_1` node, the url `127.0.0.1:8001` should be used.
For example, to send a gRPC request to the `primary_1` node, use the URL: `127.0.0.1:8001`

### Access worker node public gRPC endpoints
## Access worker node public gRPC endpoints

Similar to how someone can access the [public gRPC endpoints on a primary node](#access-primary-node-public-grpc-endpoints),
**to feed transactions** to the Narwhal cluster via the `worker` nodes could be done via the gRPC server that is
bootstrapped on the worker nodes bind to the local machine port. To send transactions the following local
Just as you access the [public gRPC endpoints on a primary node](#access-primary-node-public-grpc-endpoints), you may
similarly **feed transactions** to the Narwhal cluster via the `worker` nodes with the gRPC server
bootstrapped on the worker nodes bind to the local machine port. To send transactions, the following local
ports can be used:
* `worker_0`: 7001
* `worker_1`: 7002
Expand All @@ -110,9 +114,10 @@ ports can be used:

For example, to send a transaction to the `worker_2` node via gRPC, the url `127.0.0.1:7003` should be used.

### Folder structure
## Folder structure

Here is the Docker folder structure:

Under this folder someone will find the following
```
├── Dockerfile
├── README.md
Expand All @@ -131,45 +136,45 @@ Under this folder someone will find the following
└── entry.sh
```

Under the `validators` folder will be found the independent configuration
folder for each validator node (it is reminded that each `validator` is
constituted from one `primary` node and several `worker` nodes).
Under the `validators` folder find the independent configuration
folder for each validator node. (Remember, each `validator` is
constituted from one `primary` node and several `worker` nodes.)

The `key.json` file contains the private `key` for the corresponding node which
is associated to this node only.
The `key.json` file contains the private `key` for the corresponding node that
is associated with this node only.

The [parameters.json](validators/parameters.json) file is shared across all the nodes and contains
the core parameters for a node.

The [committee.json](validators/committee.json) file is shared across all the nodes and contains
the information about the validators (primary & worker nodes), like the public keys, addresses and
ports available etc.
the information about the validators (primary & worker nodes), like the public keys, addresses,
ports available, etc.

It has to be noted that the current docker-compose setup is mounting the [Docker/validators](validators)
folder to the service containers in order to share the folders & files in it. That allow us to experiment/change
configuration without having the need to rebuild the Docker image.
Note the current `docker-compose` setup is mounting the [Docker/validators](validators)
folder to the service containers in order to share the folders and files in it. That allows us to experiment/change
configuration without having to rebuild the Docker image.

### Docker-compose configuration
## Docker Compose configuration

The following environment variables are available to be used for each service on the
The following environment variables are available to be used for each service in the
[docker-compose.yml](docker-compose.yml) file configuration:
* `NODE_TYPE` with values `primary|worker` . Defines the node type to bootstrap
* `NODE_TYPE` with values `primary|worker`. Defines the node type to bootstrap.
* `AUTHORITY_ID` with decimal numbers, for current setup available values `0..3`. Defines the
id of the validator that the node/service corresponds to. Basically this defines which
ID of the validator that the node/service corresponds to. This defines which
configuration to use under the `validators` folder.
* `LOG_LEVEL` the level of logging for the node defined as number of `v` parameters (ex `-vvv`). The following
* `LOG_LEVEL` is the level of logging for the node defined as number of `v` parameters (ex `-vvv`). The following
levels are defined according to the number of "v"s provided: `0 | 1 => "error", 2 => "warn", 3 => "info",
4 => "debug", 5 => "trace"`.
* `CONSENSUS_DISABLED`, this value disables consensus (`Tusk`) for a primary node and enables the
`gRPC` server. The value that should be passed is `--consensus-disabled`
* `WORKER_ID` the id, as integer, for service when it runs as a worker
* `CLEANUP_DISABLED` , when provided with value `true`, it will disable the clean up of the validator folder
from the database & log data. This is useful to preserve the state between multiple docker compose runs.
* `CONSENSUS_DISABLED`. This value disables consensus (`Tusk`) for a primary node and enables the
`gRPC` server. The corresponding argument is: `--consensus-disabled`
* `WORKER_ID` is the ID, as integer, for service when it runs as a worker.
* `CLEANUP_DISABLED`, when provided with value `true`, will disable the clean up of the validator folder
from the database and log data. This is useful to preserve the state between multiple Docker Compose runs.

### Troubleshooting
## Troubleshooting

#### 1. Compile Errors when building Docker image
If come across errors while the Docker image is being build, for example errors like:
#### 1. Compile errors when building Docker image
If you encounter errors while the Docker image is being built, for example errors like:
```
error: could not compile `tonic`
#9 373.3
Expand All @@ -182,17 +187,17 @@ error: could not compile `tonic`
#9 398.4 warning: compilation terminated.
```

it is possible that the Docker engine is running out of memory and there is no capacity to properly
compile the code. In this case please try to increase the available RAM at least to 2GB and retry.
it is possible that the Docker engine is running out of memory, and there is no capacity to properly
compile the code. In this case please, increase the available RAM to at least 2GB and retry.

#### 2. Mounts denied or cannot start service errors
### 2. Mounts denied or cannot start service errors

If you try to spin up the nodes via docker-compose and you come across errors such as `mounts denied`
or `cannot start service` please make sure that you allow Docker to share your host's [Docker/validators](validators) folder
with the containers. If you are using Docker Desktop you can find more information of how to do
If you try to spin up the nodes via `docker-compose` and you come across errors such as `mounts denied`
or `cannot start service`, make sure that you allow Docker to share your host's [Docker/validators](validators) folder
with the containers. If you are using Docker Desktop, you can find more information on how to do
that here: [mac](https://docs.docker.com/desktop/mac/#file-sharing), [linux](https://docs.docker.com/desktop/linux/#file-sharing),
[windows](https://docs.docker.com/desktop/windows/#file-sharing) .

Also please check that you are not using the deprecated `devicemapper storage driver` which might also
Also, check that you are not using the deprecated `devicemapper storage driver`, which might also
cause you issues. See how to [migrate to an overlayfs driver](https://docs.docker.com/storage/storagedriver/overlayfs-driver/) .
More information about the deprecation can be found [here](https://docs.docker.com/engine/deprecated/#device-mapper-storage-driver)
More information about the deprecation can be found [here](https://docs.docker.com/engine/deprecated/#device-mapper-storage-driver)

0 comments on commit c141393

Please sign in to comment.