Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add benchmarks for a small cloud instance #1264

Merged
merged 1 commit into from
Dec 4, 2023
+81 −10
Merged

chore: add benchmarks for a small cloud instance #1264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,50 @@ Fully compatible with Redis and Memcached APIs, Dragonfly requires no code chang

## <a name="benchmarks"><a/>Benchmarks

We first compare Dragonfly with Redis on `m5.large` instance which is commonly used to run Redis
due to its single-threaded architecture. The benchmark program runs from another
load-test instance (c5n) in the same AZ using `memtier_benchmark -c 20 --test-time 100 -t 4 -d 256 --distinct-client-seed`

Dragonfly shows a comparable performance:

1. SETs (`--ratio 1:0`):

| Redis | DF |
| -----------------------------------------|----------------------------------------|
| QPS: 159K, P99.9: 1.16ms, P99: 0.82ms | QPS:173K, P99.9: 1.26ms, P99: 0.9ms |
| | |

2. GETs (`--ratio 0:1`):

| Redis | DF |
| ----------------------------------------|----------------------------------------|
| QPS: 194K, P99.9: 0.8ms, P99: 0.65ms | QPS: 191K, P99.9: 0.95ms, P99: 0.8ms |

The benchmark above shows that the algorithmic layer inside DF that allows it to scale vertically
does not take a large toll when running single-threaded.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this instance has 2 cpus right? why are u saying single threaded?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed the PR


However, if we take a bit stronger instance (m5.xlarge), the gap between DF and Redis starts growing.
(`memtier_benchmark -c 20 --test-time 100 -t 6 -d 256 --distinct-client-seed`):
1. SETs (`--ratio 1:0`):

| Redis | DF |
| ----------------------------------------|----------------------------------------|
| QPS: 190K, P99.9: 2.45ms, P99: 0.97ms | QPS: 279K , P99.9: 1.95ms, P99: 1.48 |

2. GETs (`--ratio 0:1`):

| Redis | DF |
| ----------------------------------------|----------------------------------------|
| QPS: 220K, P99.9: 0.98ms , P99: 0.8ms | QPS: 305K, P99.9: 1.03ms, P99: 0.87ms |


Dragonfly throughput capacity continues to grow with instance size,
while single-threaded Redis is bottlenecked on CPU and reaches local maxima in terms of performance.

<img src="http://static.dragonflydb.io/repo-assets/aws-throughput.svg" width="80%" border="0"/>

In benchmarks, Dragonfly showed a 25X increase in throughput compared to Redis, crossing 3.8M QPS on c6gn.16xlarge.
If we compare Dragonfly and Redis on the most network-capable instance c6gn.16xlarge,
Dragonfly showed a 25X increase in throughput compared to Redis single process, crossing 3.8M QPS.

Dragonfly's 99th percentile latency metrics at its peak throughput:

Expand Down
48 changes: 39 additions & 9 deletions docs/faq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,50 @@
# Dragonfly Frequently Asked Questions

- [What is the license model of Dragonfly? Is it an open source?](#what-is-the-license-model-of-dragonfly-is-it-an-open-source)
- [Can I use dragonfly in production?](#can-i-use-dragonfly-in-production)
- [Dragonfly provides vertical scale, but we can achieve similar throughput with X nodes in a Redis cluster.](#dragonfly-provides-vertical-scale-but-we-can-achieve-similar-throughput-with-x-nodes-in-a-redis-cluster.)
- [What if I have some commands that are missing?](#what_if_i_have_some_commands_that_are_missing)
- [Dragonfly Frequently Asked Questions](#dragonfly-frequently-asked-questions)
- [What is the license model of Dragonfly? Is it an open source?](#what-is-the-license-model-of-dragonfly-is-it-an-open-source)
- [Can I use dragonfly in production?](#can-i-use-dragonfly-in-production)
- [We benchmarked Dragonfly and we have not reached 4M qps throughput as you advertised.](#we-benchmarked-dragonfly-and-we-have-not-reached-4m-qps-throughput-as-you-advertised)
- [Dragonfly provides vertical scale, but we can achieve similar throughput with X nodes in a Redis cluster.](#dragonfly-provides-vertical-scale-but-we-can-achieve-similar-throughput-with-x-nodes-in-a-redis-cluster)
- [If only Dragonfly had this command I would use it for sure](#if-only-dragonfly-had-this-command-i-would-use-it-for-sure)


## What is the license model of Dragonfly? Is it an open source?
Dragonfly is released under [BSL 1.1](../LICENSE.md) (Business Source License). We believe that a BSL license is more permissive than AGPL-like licenses. In general terms, it means that dragonfly's code is free to use and free to change as long as you do not sell services directly related to dragonfly or in-memory datastores.
We followed the trend of other technological companies like Elastic, Redis, MongoDB, Cockroach labs, Redpanda Data to protect our rights to provide service and support for the software we are building.
Dragonfly is released under [BSL 1.1](../LICENSE.md) (Business Source License).
BSL is considered to be "source available" license and it's not strictly open-source license.
We believe that a BSL license is more permissive than licenses like AGPL, and it will allow us to
provide a competitive commercial service using our technology. In general terms,
it means that Dragonfly's code is free to use and free to change as long as you do not sell services directly related to
Dragonfly or in-memory datastores.
We followed the trend of other technological companies like Elastic, Redis, MongoDB, Cockroach labs,
Redpanda Data to protect our rights to provide service and support for the software we are building.

## Can I use dragonfly in production?
License wise you are free to use dragonfly in your production as long as you do not provide dragonfly as a managed service. From a code maturity point of view, Dragonfly's code is covered with unit testing. However as with any new software there are use cases that are hard to test and predict. We advise you to run your own particular use case on dragonfly for a few days before considering production usage.
License wise you are free to use dragonfly in your production as long as you do not provide Dragonfly as a managed service.
From a code maturity point of view, Dragonfly's code is covered with unit testing and the regression tests.
However as with any new software there are use cases that are hard to test and predict.
We advise you to run your own particular use case on dragonfly for a few days before considering production usage.

## We benchmarked Dragonfly and we have not reached 4M qps throughput as you advertised.
We conducted our experiments using a load-test generator called `memtier_benchmark`,
and we run benchmarks on AWS network-enhanced instance `c6gn.16xlarge` on recent Linux kernel versions.
Dragonfly might reach smaller throughput on other instances, but we would
still expect to reach around 1M+ qps on instances with 16-32 vCPUs.

## Dragonfly provides vertical scale, but we can achieve similar throughput with X nodes in a Redis cluster.
Dragonfly utilizes the underlying hardware in an optimal way. Meaning it can run on small 8GB instances and scale vertically to large 768GB machines with 64 cores. This versatility allows to drastically reduce complexity of running cluster workloads to a single node saving hardware resources and costs. More importantly, it reduces the complexity (total cost of ownership) of handling the multi-node cluster. In addition, Redis cluster-mode imposes some limitations on multi-key and transactional operations while Dragonfly provides the same semantics as single node Redis.
Dragonfly optimizes the use of underlying hardware, allowing it to run efficiently on instances as small as 8GB,
and scale vertically to large 2TB machines with 128 cores. This versatility significantly
reduces the complexity of running cluster workloads on a single node, saving hardware resources and costs.
More importantly, it diminishes the total cost
of ownership associated with managing multi-node clusters. In contrast, Redis in cluster
mode imposes limitations on multi-key and transactional operations, whereas Dragonfly maintains
the same semantics as a single-node Redis system.
Furthermore, scaling out horizontally with small instances can lead to instability
in production environments.
We believe that large-scale deployments of in-memory stores require both vertical and horizontal scaling,
which is not efficiently achievable with an in-memory store like Redis.

## If only Dragonfly had this command I would use it for sure
Dragonfly implements ~130 Redis commands which we think represent a good coverage of the market. However this is not based empirical data. Having said that, if you have commands that are not covered, please feel free to open an issue for that or vote for an existing issue. We will do our best to prioritise those commands according to their popularity.
Dragonfly implements ~190 Redis commands which we think represent a good coverage of the market.
However this is not based empirical data. Having said that, if you have commands that are not covered,
please feel free to open an issue for that or vote for an existing issue.
We will do our best to prioritise those commands according to their popularity.