Skip to content

Commit 3f05587

Browse files
dkechagdkechag
committed
v2.2
1 parent 635bb0d commit 3f05587

File tree

10 files changed

+366
-160
lines changed

10 files changed

+366
-160
lines changed

‎.gitignore

+22-3
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,26 @@
11
data/*.seq
2-
.DS_Store
3-
nytprof*
42
cover_db*
53
.vstags
64
/.vscode/
7-
/.idea/
5+
/.idea/
6+
MANIFEST.bak
7+
Makefile
8+
Makefile.old
9+
Build
10+
Build.bat
11+
META.*
12+
MYMETA.*
13+
.build/
14+
_build/
15+
cover_db/
16+
blib/
17+
inc/
18+
.lwpcookies
19+
.last_cover_stats
20+
.DS_Store
21+
nytprof.out
22+
nytprof/
23+
pod2htm*.tmp
24+
pm_to_blib
25+
Benchmark-DKbench-*
26+
Benchmark-DKbench-*.tar.gz

‎Changes

+7-3
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,15 @@
11
Revision history for Benchmark-DKbench
22

3+
2.2 2023-09-21
4+
- Time::Piece test is optional for all platforms.
5+
- Switched to monotonic clock instead of wall.
6+
- dkbench -h option.
7+
38
2.1 2023-09-09
49
- Added Text::Levenshtein and DBI/SQL benchmarks.
5-
- Added -ver command line option for dkbench to filter benchmarks.
10+
- Added -ver command line option.
611
- Made tests locale-independent.
7-
- Switched to System::Info from Sys::Info
12+
- Switched to System::Info from Sys::Info.
813
- Removed libjpeg dependency.
914
- More documentation.
1015

@@ -14,4 +19,3 @@ Revision history for Benchmark-DKbench
1419

1520
1.0 2021-12-21
1621
- First version.
17-

‎MANIFEST

-1
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,6 @@ README
7373
setup_dkbench
7474
t/00-load.t
7575
t/cover.t
76-
t/cpu.t
7776
t/simple.t
7877
xt/author/manifest.t
7978
xt/author/pod.t

‎Makefile.PL

+2-2
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,7 @@ my %WriteMakefileArgs = (
2525
},
2626
EXE_FILES => ["dkbench", "setup_dkbench"],
2727
TEST_REQUIRES => {
28-
'Test2::Tools::Exception' => '0',
29-
'Test2::V0' => '0',
28+
'Test2::V0' => '0',
3029
},
3130
PREREQ_PM => {
3231
'Astro::Coord::Constellations' => '0',
@@ -49,6 +48,7 @@ my %WriteMakefileArgs = (
4948
'MCE::Loop' => '0',
5049
'SQL::Inserter' => '0',
5150
'SQL::Abstract::Classic' => '0',
51+
'System::CPU' => '0',
5252
'System::Info' => '0',
5353
'Test::Harness' => '0',
5454
'Test::Requires' => '0',

‎README.md

+265
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,265 @@
1+
# NAME
2+
3+
Benchmark::DKbench - Perl CPU Benchmark
4+
5+
# SYNOPSIS
6+
7+
# Run the suite single-threaded and then multi-threaded on multi-core systems
8+
# Will print scores for the two runs and multi/single thread scalability
9+
dkbench
10+
11+
# A dual-thread "quick" run (with times instead of scores)
12+
dkbench -j 2 -q
13+
14+
# If BioPerl is installed, enable the BioPerl benchmarks by downloading Genbank data
15+
dkbench --setup
16+
17+
# Force install the reference versions of all CPAN modules
18+
setup_dkbench --force
19+
20+
# DESCRIPTION
21+
22+
A Perl benchmark suite for general compute, created to evaluate the comparative
23+
performance of systems when running computationally intensive Perl (both pure Perl
24+
and C/XS) workloads. It is a good overall indicator for generic CPU performance in
25+
real-world scenarios. It runs single and multi-threaded (able to scale to hundreds
26+
of CPUs) and can be fully customized to run the benchmarks that better suit your own
27+
scenario.
28+
29+
# INSTALLATION
30+
31+
The only non-CPAN software required to install/run the suite is a build environment
32+
for the C/XS modules (C compiler, make etc.) and Perl. On the most popular Linux
33+
package managers you can easily set up such an environment (as root or with sudo):
34+
35+
# Debian/Ubuntu etc
36+
apt-get update
37+
apt-get install build-essential perl cpanminus
38+
39+
# CentOS/Red Hat
40+
yum update
41+
yum install gcc make patch perl perl-App-cpanminus
42+
43+
After that, you can use [App::cpanminus](https://metacpan.org/pod/App%3A%3Acpanminus) to install the benchmark suite (as
44+
root/sudo is the easiest, will install for all users):
45+
46+
cpanm -n Benchmark::DKbench
47+
48+
See the `setup_dkbench` script below for more on the installation of a couple of
49+
optional benchmarks and standardizing your benchmarking environment.
50+
51+
# SCRIPTS
52+
53+
You will most likely only ever need the main script `dkbench` which launches the
54+
suite, although `setup_dkbench` can help with setup or standardizing/normalizing your
55+
benchmarking environment.
56+
57+
## `dkbench`
58+
59+
The main script that runs the DKbench benchmark suite. If [BioPerl](https://metacpan.org/pod/BioPerl) is installed,
60+
you may want to start with `dkbench --setup`. But beyond that, there are many
61+
options to control number of threads, iterations, which benchmarks to run etc:
62+
63+
dkbench [options]
64+
65+
Options:
66+
--threads <i>, -j <i> : Number of benchmark threads (default is 1).
67+
--multi, -m : Multi-threaded using all your CPU cores/threads.
68+
--max_threads <i> : Override the cpu detection to specify max cpu threads.
69+
--iter <i>, -i <i> : Number of suite iterations (with min/max/avg at the end).
70+
--include <regex> : Run only benchmarks that match regex.
71+
--exclude <regex> : Do not run benchmarks that match regex.
72+
--time, -t : Report time (sec) instead of score.
73+
--quick, -q : Quick benchmark run (implies -t).
74+
--no_mce : Do not run under MCE::Loop (implies -j 1).
75+
--skip_bio : Skip BioPerl benchmarks.
76+
--skip_prove : Skip Moose prove benchmark.
77+
--time_piece : Run optional Time::Piece benchmark (see benchmark details).
78+
--bio_codons : Run optional BioPerl Codons benchmark (does not scale well).
79+
--sleep <i> : Sleep for <i> secs after each benchmark.
80+
--setup : Download the Genbank data to enable the BioPerl tests.
81+
--datapath <path> : Override the path where the expected benchmark data is found.
82+
--ver <num> : Skip benchmarks added after the specified version.
83+
--help -h : Show basic help and exit.
84+
85+
The default run (no options) will run all the benchmarks both single-threaded and
86+
multi-threaded (using all detected CPU cores/hyperthreads) and show you scores and
87+
multi vs single threaded scalability.
88+
89+
The scores are calibrated such that a reference CPU (Intel Xeon Platinum 8481C -
90+
Sapphire Rapids) would achieve a score of 1000 in a single-core benchmark run using
91+
the default software configuration (Linux/Perl 5.36.0 with reference CPAN modules).
92+
93+
The multi-thread scalability should approach 100% if each thread runs on a full core
94+
(i.e. no SMT), and the core can maintain the clock speed it had on the single-thread
95+
runs. Note that the overall scalability is an average of the benchmarks that drops
96+
non-scaling outliers (over 2\*stdev less than the mean).
97+
98+
The suite will report a Pass/Fail per benchmark. A failure may be caused if you have
99+
different CPAN module version installed - this is normal, and you will be warned.
100+
101+
The suite uses [MCE::Loop](https://metacpan.org/pod/MCE%3A%3ALoop) to run on the desired number of parallel threads, although
102+
there is an option to disable it, which forces a single-thread run.
103+
104+
## `setup_dkbench`
105+
106+
Simple installer to check/get the reference versions of CPAN modules and download
107+
the Genbank data file required for the BioPerl benchmarks of the DKbench suite.
108+
109+
It assumes that you have some software already installed (see INSTALLATION above),
110+
try `setup_dkbench --help` will give you more details.
111+
112+
setup_dkbench [--force --sudo --test --data=s --help]
113+
114+
Options:
115+
--sudo : Will use sudo for cpanm calls.
116+
--force : Will install reference CPAN module versions and re-download the genbank data.
117+
--test : Will run the test suites for the CPAN module (default behaviour is to skip).
118+
--data=s : Data dir path to copy files from. Should not need if you installed DKbench.
119+
--help : Print this help text and exit.
120+
121+
Running it without any options will fetch the data for the BioPerl tests (similar to
122+
`dkbench --setup`) and use `cpanm` to install any missing libraries.
123+
124+
Using it with `--force` will install the reference CPAN module versions, including
125+
BioPerl which is not a requirement for DKbench, but enables the BioPerl benchmarks.
126+
127+
The reference Perl and CPAN versions are suggested if you want a fair comparison
128+
between systems and also for the benchmark Pass/Fail results to be reliable.
129+
130+
# BENCHMARKS
131+
132+
The suite consists of 21 benchmarks, 19 will run by default. However, the
133+
`BioPerl Monomers` requires the optional [BioPerl](https://metacpan.org/pod/BioPerl) to be installed and Genbank
134+
data to be downloaded (`dkbench --setup` can do the latter), so you will only
135+
see 18 benchmarks running just after a standard install. Because the overall score
136+
is an average, it is generally unaffected by adding or skipping a benchmark or two.
137+
138+
The optional benchmarks are enabled with the `--time_piece` and `--bio_codons`
139+
options.
140+
141+
- `Astro` : Calculates precession between random epochs and finds the
142+
constellation for random equatorial coordinates using [Astro::Coord::Precession](https://metacpan.org/pod/Astro%3A%3ACoord%3A%3APrecession)
143+
and [Astro::Coord::Constellations](https://metacpan.org/pod/Astro%3A%3ACoord%3A%3AConstellations) respectively.
144+
- `BioPerl Codons` : Counts codons on a sample bacterial sequence. Requires
145+
[BioPerl](https://metacpan.org/pod/BioPerl) to be installed.
146+
This test does not scale well on multiple threads, so is disabled by default (use
147+
`--bio_codons`) option. Requires data fetched using the `--setup` option.
148+
- `BioPerl Monomers` : Counts monomers on 500 sample bacterial sequences using
149+
[BioPerl](https://metacpan.org/pod/BioPerl) (which needs to be installed). Requires data fetched using the `--setup`
150+
option.
151+
- `CSS::Inliner` : Inlines CSS on 2 sample wiki pages using [CSS::Inliner](https://metacpan.org/pod/CSS%3A%3AInliner).
152+
- `Crypt::JWT` : Creates large JSON Web Tokens with RSA and EC crypto keys
153+
using [Crypt::JWT](https://metacpan.org/pod/Crypt%3A%3AJWT).
154+
- `DateTime` : Creates and manipulates [DateTime](https://metacpan.org/pod/DateTime) objects.
155+
- `DBI/SQL` : Creates a mock [DBI](https://metacpan.org/pod/DBI) connection (using [DBD::Mock](https://metacpan.org/pod/DBD%3A%3AMock)) and passes
156+
it insert/select statements using [SQL::Inserter](https://metacpan.org/pod/SQL%3A%3AInserter) and [SQL::Abstract::Classic](https://metacpan.org/pod/SQL%3A%3AAbstract%3A%3AClassic).
157+
The latter is quite slow at creating the statements, but it is widely used.
158+
- `Digest` : Creates MD5, SH1 and SHA-512 digests of a large string.
159+
- `Encode` : Encodes/decodes large strings from/to UTF-8/16, cp-1252.
160+
- `HTML::FormatText` : Converts HTML to text for 2 sample wiki pages using
161+
[HTML::FormatText](https://metacpan.org/pod/HTML%3A%3AFormatText).
162+
- `Imager` : Loads a sample image and performs edits/manipulations with
163+
[Imager](https://metacpan.org/pod/Imager), including filters like gaussian, unsharp mask, mandelbrot.
164+
- `JSON::XS` : Encodes/decodes random data structures to/from JSON using
165+
[JSON::XS](https://metacpan.org/pod/JSON%3A%3AXS).
166+
- `Math::DCT` : Does 8x8, 18x18 and 32x32 DCT transforms with [Math::DCT](https://metacpan.org/pod/Math%3A%3ADCT).
167+
- `Math::MatrixReal` : Performs various manipulations on [Math::MatrixReal](https://metacpan.org/pod/Math%3A%3AMatrixReal)
168+
matrices.
169+
- `Moose` : Creates [Moose](https://metacpan.org/pod/Moose) objects.
170+
- `Moose prove` : Runs 110 tests from the Moose 2.2201 test suite. The least
171+
CPU-intensive test (which is why there is the option `--no_prove` to disable it),
172+
most of the time will be spent loading the interpreter and the Moose module for each
173+
test, which is behaviour representative of how a Perl test suite runs by default.
174+
- `Primes` : Calculates all primes up to 7.5 million. Small number with
175+
repeat was chosen to keep low memory (this is a pure Perl function no Math libraries).
176+
- `Regex/Subst` : Concatenates 3 wiki pages into a byte string then matches
177+
3 typical regexes (for names, emails, URIs), replaces html tags with their contents
178+
(starting with the innermost) and does calls subst a few times.
179+
- `Regex/Subst utf8` : Exactly the same as `Regex/Subst`, but reads into
180+
a utf8 string. Perl version can make a big difference, as Unicode behaviour has
181+
changed (old Perl versions are faster but less strict in general).
182+
- `Text::Levenshtein` : The edit distance for strings of various lengths (up
183+
to 2500) are calculated using [Text::Levenshtein::XS](https://metacpan.org/pod/Text%3A%3ALevenshtein%3A%3AXS) and [Text::Levenshtein::Damerau::XS](https://metacpan.org/pod/Text%3A%3ALevenshtein%3A%3ADamerau%3A%3AXS).
184+
- `Time::Piece` : Creates and manipulates/converts Time::Piece objects. It
185+
is disabled by default because it uses the OS time libraries, so it might skew results
186+
if you are trying to compare CPUs on different OS platforms. It can be enabled with
187+
the `--time_piece` option. For MacOS specifically, it can only be enabled if `--no_mce`
188+
is specified, as it runs extremely slow when forked.
189+
190+
# EXPORTED FUNCTIONS
191+
192+
You will normally not use the Benchmark::DKbench module itself, but here are the
193+
exported functions that the `dkbench` script uses for reference:
194+
195+
## `system_identity`
196+
197+
my $cores = system_identity();
198+
199+
Prints out software/hardware configuration and returns then number of cores detected.
200+
201+
## `suite_run`
202+
203+
my %stats = suite_run(\%options);
204+
205+
Runs the benchmark suite given the `%options` and prints results. Returns a hash
206+
with run stats.
207+
208+
## `calc_scalability`
209+
210+
calc_scalability(\%options, \%stat_single, \%stat_multi);
211+
212+
Given the `%stat_single` results of a single-threaded `suite_run` and `%stat_multi`
213+
results of a multi-threaded run, will calculate and print the multi-thread scalability.
214+
215+
# NOTES
216+
217+
The benchmark suite was created to compare the performance of various cloud offerings.
218+
You can see the [original perl blog post](http://blogs.perl.org/users/dimitrios_kechagias/2022/03/cloud-provider-performance-comparison-gcp-aws-azure-perl.html)
219+
as well as the [2023 follow-up](https://dev.to/dkechag/cloud-vm-performance-value-comparison-2023-perl-more-1kpp).
220+
221+
The benchmarks for the first version were more tuned to what I would expect to run
222+
on the servers I was testing, in order to choose the optimal types for the company
223+
I was working for. The second version has expanded a bit over that, and is friendlier
224+
to use.
225+
226+
Althought this benchmark is in general a good indicator of general CPU performance
227+
and can be customized to your needs, no benchmark is as good as running your own
228+
actual workload.
229+
230+
## SCORES
231+
232+
Some sample DKbench score results from various systems for comparison (all on
233+
reference setup with Perl 5.36.0):
234+
235+
CPU Cores/HT Single Multi Scalability
236+
Intel i7-4750HQ @ 2.0 (MacOS) 4/8 612 2332 46.9%
237+
AMD Ryzen 5 PRO 4650U @ 2.1 (WSL) 6/12 905 4444 40.6%
238+
Apple M1 Pro @ 3.2 (MacOS) 10/10 1283 10026 78.8%
239+
Apple M2 Pro @ 3.5 (MacOS) 12/12 1415 12394 73.1%
240+
Ampere Altra @ 3.0 (Linux) 48/48 708 32718 97.7%
241+
Intel Xeon Platinum 8481C @ 2.7 (Linux) 88/176 1000 86055 48.9%
242+
AMD EPYC Milan 7B13 @ 2.45 (Linux) 112/224 956 104536 49.3%
243+
AMD EPYC Genoa 9B14 @ 2.7 (Linux) 180/360 1197 221622 51.4%
244+
245+
# AUTHOR
246+
247+
Dimitrios Kechagias, `<dkechag at cpan.org>`
248+
249+
# BUGS
250+
251+
Please report any bugs or feature requests either on [GitHub](https://github.com/dkechag/Benchmark-DKbench) (preferred), or on RT (via the email
252+
`bug-Benchmark-DKbench at rt.cpan.org` or [web interface](https://rt.cpan.org/NoAuth/ReportBug.html?Queue=Benchmark-DKbench)).
253+
254+
I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
255+
256+
# GIT
257+
258+
[https://github.com/dkechag/Benchmark-DKbench](https://github.com/dkechag/Benchmark-DKbench)
259+
260+
# LICENSE AND COPYRIGHT
261+
262+
This software is copyright (c) 2021-2023 by Dimitrios Kechagias.
263+
264+
This is free software; you can redistribute it and/or modify it under
265+
the same terms as the Perl 5 programming language system itself.

0 commit comments

Comments
 (0)