Skip to content

Commit 9e6450c

Browse files
jdjaustinjdjaustin
authored
Generate Daily Benchmark Results (#2367)
1 parent 8b25f25 commit 9e6450c

9 files changed

+838
-260
lines changed

‎.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,7 @@ genesis.json
1414
node/rpc-api/static/ast.ron
1515
documentation/book/**/*
1616
node_modules
17+
/scripts/benchmark_db/.bundle/
18+
/scripts/benchmark_db/vendor/
19+
/scripts/benchmark_db/*.md
20+
/scripts/benchmark_db/*.csv

‎scripts/benchmark_db/.rubocop.yml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
AllCops:
2+
NewCops: enable
3+
Layout/LineLength:
4+
Max: 120
5+
Metrics/MethodLength:
6+
Max: 20

‎scripts/benchmark_db/Gemfile

+9
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,14 @@ source 'https://rubygems.org'
44

55
git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
66

7+
gem 'concurrent-ruby', '~> 1.2', '>= 1.2.2'
8+
gem 'csv', '~> 3.2', '>= 3.2.6'
79
gem 'deep_merge', '~> 1.2', '>= 1.2.2'
10+
gem 'fileutils', '~> 1.7'
11+
gem 'logger', '~> 1.5', '>= 1.5.3'
12+
gem 'open3', '~> 0.1.2'
13+
gem 'optparse', '~> 0.3.1'
14+
gem 'set', '~> 1.0', '>= 1.0.3'
15+
gem 'tmpdir', '~> 0.1.3'
816
gem 'toml-rb', '~> 2.2'
17+
gem 'tomlrb', '~> 2.0', '>= 2.0.3'

‎scripts/benchmark_db/Gemfile.lock

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
GEM
2+
remote: https://rubygems.org/
3+
specs:
4+
citrus (3.0.2)
5+
concurrent-ruby (1.2.2)
6+
csv (3.2.6)
7+
deep_merge (1.2.2)
8+
fileutils (1.7.0)
9+
logger (1.5.3)
10+
open3 (0.1.2)
11+
optparse (0.3.1)
12+
set (1.0.3)
13+
tmpdir (0.1.3)
14+
fileutils
15+
toml-rb (2.2.0)
16+
citrus (~> 3.0, > 3.0)
17+
tomlrb (2.0.3)
18+
19+
PLATFORMS
20+
arm64-darwin-21
21+
22+
DEPENDENCIES
23+
concurrent-ruby (~> 1.2, >= 1.2.2)
24+
csv (~> 3.2, >= 3.2.6)
25+
deep_merge (~> 1.2, >= 1.2.2)
26+
fileutils (~> 1.7)
27+
logger (~> 1.5, >= 1.5.3)
28+
open3 (~> 0.1.2)
29+
optparse (~> 0.3.1)
30+
set (~> 1.0, >= 1.0.3)
31+
tmpdir (~> 0.1.3)
32+
toml-rb (~> 2.2)
33+
tomlrb (~> 2.0, >= 2.0.3)
34+
35+
BUNDLED WITH
36+
2.4.6

‎scripts/benchmark_db/README.md

+91-24
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,116 @@
11
# Forest benchmark db script
22

3-
This script is here to help with testing of Forest db backends and their
4-
configuration.
3+
This script was developed to help with testing of Forest db backends and their
4+
configuration; the script now also allows benchmarking ("daily" benchmarks) of
5+
Forest and Lotus snapshot import times (in sec) and validation times (in
6+
tipsets/sec).
57

68
## Install dependencies
79

8-
You will need to install Ruby first. Then go into `benchmark_db` and type:
10+
[Install Ruby](https://www.ruby-lang.org/en/documentation/installation/) first.
11+
Then go into `scripts/benchmark_db` and execute the following commands:
912

1013
```
14+
$ bundle config set --local path 'vendor/bundle'
1115
$ bundle install
1216
```
1317

18+
Note: depending upon your Ruby installation, it may be necessary to execute
19+
`gem install bundler` first. In case of any issues with "native extensions"
20+
during `bundle install` on a \*nix machine, it may also be necessary to execute
21+
`apt-get update && apt-get install -y build-essential ruby-dev`.
22+
23+
The daily benchmarks also require the installation of
24+
[aria2](https://github.com/aria2/aria2) and
25+
[zstd](https://github.com/facebook/zstd), as well as dependencies required for
26+
the installation of [Forest](https://github.com/ChainSafe/forest) and
27+
[Lotus](https://github.com/filecoin-project/lotus) (note that the script handles
28+
installation of the Forest and Lotus binaries).
29+
1430
## Run benchmarks
1531

16-
You need to run the script at the root of the repository. Ie:
32+
Run the script at the root of the repository. I.e.,:
1733

1834
```
19-
$ ./benchmark_db/bench.rb 2369040_2022_11_25t12_00_00z.car
35+
$ ./scripts/benchmark_db/bench.rb <path to snapshot> <optional flags>
2036
```
2137

22-
You can create a selection of benchmarks using the `--pattern` flag. If used in
23-
conjunction with `--dry-run` you will see what commands will be run:
38+
If the user does not specify a path to a snapshot, the script will automatically
39+
download a fresh snapshot, then pause for 5 minutes to allow the network to
40+
advance to ensure that enough time will be spent in the `message sync` stage for
41+
proper calculation of the validation time metric. Also note that if `--chain` is
42+
specified, the user must provide a script matching the specified `<chain>` (the
43+
script defaults to `mainnet`, so if `--chain` is not specified, provide a
44+
`mainnet` snapshot).
45+
46+
If the `--daily` flag is included in the command line arguments, the script will
47+
run the daily benchmarks specified earlier; otherwise the script will run the
48+
backend metrics.
49+
50+
On many machines, running the script with `--chain mainnet` may require more
51+
space than allocated to the `tmp` partition. To address this, specify the
52+
`--tempdir` flag with a user-defined directory (which will automatically be
53+
created if it does not already exist).
54+
55+
To create a selection of benchmarks, use the `--pattern` flag (current defined
56+
patterns are `'*'`, `'baseline'`, `'jemalloc'`, and `'mimalloc'`). Using
57+
`--dry-run` outputs to the terminal the commands the script will run (without
58+
actually running the commands):
2459

2560
```
26-
$ ./scripts/benchmark_db/bench.rb 2369040_2022_11_25t12_00_00z.car --pattern 'baseline*,paritydb' --dry-run
27-
Running bench: baseline
61+
$ ./scripts/benchmark_db/bench.rb <path to snapshot> --chain calibnet --pattern jemalloc --dry-run
62+
(I) Using snapshot: <path to snapshot>
63+
(I) WORKING_DIR: <generated directory>
64+
65+
(I) Running bench: jemalloc
66+
(I) Building artefacts...
67+
(I) Cloning repository
68+
$ git clone https://github.com/ChainSafe/forest.git forest
69+
(I) Clean and build client
2870
$ cargo clean
29-
$ cargo build --release
30-
$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
31-
$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load --height 2368640
32-
Wiping db
71+
$ cargo build --release --no-default-features --features forest_fil_cns,jemalloc
72+
$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
73+
$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load=true --height <height>
74+
(I) Clean db
75+
$ ./forest/target/release/forest-cli -c <path>
76+
toml db clean --force
3377
34-
Running bench: baseline-with-stats
78+
(I) Wrote result_<time>.md
79+
```
80+
81+
```
82+
$ ./scripts/benchmark_db/bench.rb <path to snapshot> --chain calibnet --dry-run --daily
83+
(I) Using snapshot: <path to snapshot>
84+
(I) WORKING_DIR: <generated directory>
85+
86+
(I) Running bench: forest
87+
(I) Building artefacts...
88+
(I) Cloning repository
89+
$ git clone https://github.com/ChainSafe/forest.git forest
90+
(I) Clean and build client
3591
$ cargo clean
3692
$ cargo build --release
37-
$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
38-
$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load --height 2368640
39-
Wiping db
93+
$ ./forest/target/release/forest-cli fetch-params --keys
94+
$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
95+
$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false
96+
(I) Clean db
97+
$ ./forest/target/release/forest-cli -c <path> db clean --force
4098
41-
Running bench: paritydb
42-
$ cargo clean
43-
$ cargo build --release --no-default-features --features forest_fil_cns,paritydb
44-
$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
45-
$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load --height 2368640
46-
Wiping db
99+
(I) Running bench: lotus
100+
(I) Building artefacts...
101+
(I) Cloning repository
102+
$ git clone https://github.com/filecoin-project/lotus.git lotus
103+
(I) Clean and build client
104+
$ make clean
105+
$ make calibnet
106+
$ ./lotus/lotus daemon --import-snapshot <tbd> --halt-after-import
107+
$ ./lotus/lotus daemon
108+
(I) Clean db
109+
110+
Wrote result_<time>.csv
47111
```
48112

49-
Benchmark results will be written to a markdown file at the end.
113+
As seen in these examples, if `--daily` is passed in the command line, daily
114+
benchmark results are written to a CSV in the current directory with naming
115+
format `result_<time>.csv`. Otherwise, backend benchmark results will be written
116+
to a markdown file with a similar naming convention.

0 commit comments

Comments
 (0)