[perf] Improve bench, and tweak db options #818

gdanezis · 2022-03-13T23:49:26Z

This PR contains the reasonable changes I made pre-GDC relating to tweaking the DB and improving the bench and perf:

Added some parallelism to the bench setup, and now using rayon to create transactions / certificates and seed the DB on all threads.
Exposed per-table/cf options, and tweaked some tables for more efficient point lookups
Now create locks when we insert object (unrelated but nice)

On this PR on AWS (c5d.metal 48 physical cores) we do:

$ cargo run --release --bin=bench -- --num-accounts 480000 --max-in-flight 48000 --use-move --tcp-connections 48 --db-dir /dataother/db --db-cpus 4 --send-timeout-us 20000000
> Total time: 12916005us, items: 480000, tx/sec: 37163.194037165515

$ cargo run --release --bin=bench -- --num-accounts 480000 --max-in-flight 48000 --use-move --tcp-connections 96 --db-dir /dataother/db --db-cpus 4 --send-timeout-us 20000000
> Total time: 11532369us, items: 480000, tx/sec: 41621.977236420375

$ cargo run --release --bin=bench -- --num-accounts 480000 --max-in-flight 96000 --use-move --tcp-connections 96 --db-dir /dataother/db --db-cpus 4 --send-timeout-us 20000000
> Total time: 11342484us, items: 480000, tx/sec: 42318.77250168481

The main branch baseline is:

$ cargo run --release --bin=bench -- --num-accounts 480000 --max-in-flight 96000 --use-move --tcp-connections 96 --db-dir /dataother/db --db-cpus 4 --send-timeout-us 20000000
> Total time: 13487272us, items: 480000, tx/sec: 35589.10949523373

And it is painfully slow to launch since the computation of certs / txs takes much longer than the bench.

asonnino

Left a few comments.
Are we sure we are not overfitting for a specific AWS instance?

asonnino · 2022-03-16T10:05:52Z

sui/src/bench.rs

@@ -158,24 +159,23 @@ impl ClientServerBenchmark {

        let mut opts = Options::default();
        opts.increase_parallelism(self.db_cpus as i32);
-        let store = Arc::new(AuthorityStore::open(path, None));
+        opts.set_write_buffer_size(256 * 1024 * 1024);


Why these numbers specifically?

The default is 64Mb and even my laptop has way more memory. Since the DB is currently our own store, we allow the memtables to be a little bigger to accommodate spikes.

asonnino · 2022-03-16T10:07:23Z

sui/src/bench.rs

+            let state = AuthorityState::new(
+                committee.clone(),
+                public_auth0,
+                Arc::pin(secret_auth0),
+                store,
+                genesis::clone_genesis_compiled_modules(),
+                &mut genesis::get_genesis_context(),
+            )
+            .await;
+
            state


Suggested change

let state = AuthorityState::new(

committee.clone(),

public_auth0,

Arc::pin(secret_auth0),

store,

genesis::clone_genesis_compiled_modules(),

&mut genesis::get_genesis_context(),

)

.await;

state

AuthorityState::new(

committee.clone(),

public_auth0,

Arc::pin(secret_auth0),

store,

genesis::clone_genesis_compiled_modules(),

&mut genesis::get_genesis_context(),

)

.await

Thanks, changed that!

asonnino · 2022-03-16T10:33:35Z

sui_core/src/authority/authority_store.rs

@@ -20,6 +22,8 @@ pub type AuthorityStore = SuiDataStore<true>;
 #[allow(dead_code)]
 pub type ReplicaStore = SuiDataStore<false>;

+const NUM_SHARDS: usize = 2048;


Is this optimised for the m5d.metal AWS machine?

This is also used locally, and we have not optimized it much to be honest. Since we process (on laptop) about 10K-20K per second, I thought I should up this a little to reduce contention.

asonnino · 2022-03-16T10:37:08Z

And it is painfully slow to launch since the computation of certs / txs takes much longer than the bench.

When possible I will add a benchmark client like this one: https://github.com/asonnino/fastpay/blob/extensions/benchmark_client/src/client.rs

…s#859 This enacts the reversion of MystenLabs#818

This enacts the reversion of #818

* Update narwhal pointer * fix: adapt the Narwhal pointer in Sui post merge of narwhal/#859 (#4354) This enacts the reversion of #818 * Add comments from #4219 * Update authority.rs after rebase * hakari generate Co-authored-by: François Garillot <4142+huitseeker@users.noreply.github.com>

Batch execution with single execution adapter

…enLabs#818)

gdanezis added the perf optimization label Mar 14, 2022

gdanezis mentioned this pull request Mar 14, 2022

[authority] More straight forward idempotency and cleanup of TODOs #772

Merged

asonnino self-requested a review March 16, 2022 10:00

asonnino approved these changes Mar 16, 2022

View reviewed changes

gdanezis force-pushed the db-perf-reasonable branch from d5197e9 to e7821af Compare March 16, 2022 13:44

George Danezis added 7 commits March 16, 2022 15:09

Imporve bench, and tweak db options

58113f0

Make fmt happy

536073f

Pass objects from bench by ref

af759cb

Fix comments

347ff76

Trigger

e7bd557

Info

afe5f88

fmt

2b3fbf0

gdanezis force-pushed the db-perf-reasonable branch from a5f5e4b to 2b3fbf0 Compare March 16, 2022 15:09

gdanezis merged commit 0d2032d into main Mar 16, 2022

gdanezis deleted the db-perf-reasonable branch March 16, 2022 16:16

huitseeker added a commit to huitseeker/sui that referenced this pull request Aug 30, 2022

fix: adapt the Narwhal pointer in Sui post merge of narwhal/MystenLab…

1543b92

…s#859 This enacts the reversion of MystenLabs#818

arun-koshy pushed a commit that referenced this pull request Aug 30, 2022

fix: adapt the Narwhal pointer in Sui post merge of narwhal/#859 (#4354)

29f25fe

This enacts the reversion of #818

mwtian pushed a commit that referenced this pull request Sep 12, 2022

Batch execution with single execution adapter (#818)

4ef35a5

Batch execution with single execution adapter

mwtian pushed a commit that referenced this pull request Sep 12, 2022

revert: 4ef35a5 - Batch execution with single execution adapter (#818)

50d1631

mwtian pushed a commit to mwtian/sui that referenced this pull request Sep 29, 2022

Batch execution with single execution adapter (MystenLabs#818)

017d467

Batch execution with single execution adapter

mwtian pushed a commit to mwtian/sui that referenced this pull request Sep 29, 2022

revert: 017d467 - Batch execution with single execution adapter (Myst…

5123125

…enLabs#818)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[perf] Improve bench, and tweak db options #818

[perf] Improve bench, and tweak db options #818

gdanezis commented Mar 13, 2022 •

edited

Loading

asonnino left a comment

asonnino Mar 16, 2022

gdanezis Mar 16, 2022

asonnino Mar 16, 2022

gdanezis Mar 16, 2022

asonnino Mar 16, 2022

gdanezis Mar 16, 2022

asonnino commented Mar 16, 2022

[perf] Improve bench, and tweak db options #818

[perf] Improve bench, and tweak db options #818

Conversation

gdanezis commented Mar 13, 2022 • edited Loading

asonnino left a comment

Choose a reason for hiding this comment

asonnino Mar 16, 2022

Choose a reason for hiding this comment

gdanezis Mar 16, 2022

Choose a reason for hiding this comment

asonnino Mar 16, 2022

Choose a reason for hiding this comment

gdanezis Mar 16, 2022

Choose a reason for hiding this comment

asonnino Mar 16, 2022

Choose a reason for hiding this comment

gdanezis Mar 16, 2022

Choose a reason for hiding this comment

asonnino commented Mar 16, 2022

gdanezis commented Mar 13, 2022 •

edited

Loading