Reworked bench.py/bench_runner/how bench measurements are recorded

This is based on how bench.py/bench_runners have actually been used in
practice. The main changes have been to make the output of bench.py more
readibly consumable by plot.py/plotmpl.py without needing a bunch of
hacky intermediary scripts.

Now instead of a single per-bench BENCH_START/BENCH_STOP, benches can
have multiple named BENCH_START/BENCH_STOP invocations to measure
multiple things in one run:

  BENCH_START("fetch", i, STEP);
  lfsr_rbyd_fetch(&lfs, &rbyd_, rbyd.block, CFG->block_size) => 0;
  BENCH_STOP("fetch");

Benches can also now report explicit results, for non-io measurements:

  BENCH_RESULT("usage", i, STEP, rbyd.eoff);

The extra iter/size parameters to BENCH_START/BENCH_RESULT also allow
some extra information to be calculated post-bench. This infomation gets
tagged with an extra bench_agg field to help organize results in
plot.py/plotmpl.py:

  - bench_meas=<meas>+amor, bench_agg=raw - amortized results
  - bench_meas=<meas>+div,  bench_agg=raw - per-byte results
  - bench_meas=<meas>+avg,  bench_agg=avg - average over BENCH_SEED
  - bench_meas=<meas>+min,  bench_agg=min - minimum over BENCH_SEED
  - bench_meas=<meas>+max,  bench_agg=max - maximum over BENCH_SEED

---

Also removed all bench.tomls for now. This may seem counterproductive in
a commit to improve benchmarking, but I'm not sure there's actual value
to keeping bench cases committed in tree.

These were alway quick to fall out of date (at the time of this commit
most of the low-level bench.tomls, rbyd, btree, etc, no longer
compiled), and most benchmarks were one-off collections of scripts/data
with results too large/cumbersome to commit and keep updated in tree.

I think the better way to approach benchmarking is a seperate repo
(multiple repos?) with all related scripts/state/code and results
committed into a hopefully reproducible snapshot. Keeping the
bench.tomls in that repo makes more sense in this model.

There may be some value to having benchmarks in CI in the future, but
for that to make sense they would need to actually fail on performance
regression. How to do that isn't so clear. Anyways we can always address
this in the future rather than now.
This commit is contained in:
Christopher Haster
2023-11-03 10:27:17 -05:00
parent 4069cf5701
commit e8bdd4d381
11 changed files with 272 additions and 1399 deletions
-113
View File
@@ -1,113 +0,0 @@
# Bench our mid-level B-trees
after = 'bench_rbyd'
# maximize lookahead buffer, we don't actually gc so we only get one pass
# of the disk for these tests
defines.LOOKAHEAD_SIZE = 'BLOCK_COUNT / 8'
[cases.bench_btree_lookup]
defines.N = [8, 16, 32, 64, 128, 256, 1024]
# 0 = in-order
# 1 = reversed-order
# 2 = random-order
defines.ORDER = [0, 1, 2]
defines.SEED = 42
in = 'lfs.c'
code = '''
lfs_t lfs;
lfs_init(&lfs, CFG) => 0;
// create free lookahead
memset(lfs.lookahead.buffer, 0, CFG->lookahead_size);
lfs.lookahead.start = 0;
lfs.lookahead.size = lfs_min(8*CFG->lookahead_size,
CFG->block_count);
lfs.lookahead.next = 0;
lfs_alloc_ack(&lfs);
uint32_t prng = SEED;
// create a tree with N elements
lfsr_btree_t btree = LFSR_BTREE_NULL;
const char *alphas = "abcdefghijklmnopqrstuvwxyz";
for (lfs_size_t i = 0; i < N; i++) {
lfs_off_t i_
= (ORDER == 0) ? i
: (ORDER == 1) ? 0
: BENCH_PRNG(&prng) % (lfsr_btree_weight(&btree)+1);
lfsr_btree_push(&lfs, &btree, i_, LFSR_TAG_INLINED, 1,
LFSR_DATA_BUF(&alphas[i % 26], 1)) => 0;
}
// assume an unfetched btree
btree.root.off = 0;
// bench lookup
BENCH_START();
lfs_size_t i = BENCH_PRNG(&prng) % N;
uint8_t buffer[4];
lfsr_tag_t tag_;
lfs_size_t weight_;
lfsr_btree_get(&lfs, &btree, i,
&tag_, &weight_, buffer, 4) => 1;
assert(tag_ == LFSR_TAG_INLINED);
assert(weight_ == 1);
BENCH_STOP();
'''
[cases.bench_btree_commit]
defines.N = [8, 16, 32, 64, 128, 256, 1024]
# 0 = in-order
# 1 = reversed-order
# 2 = random-order
defines.ORDER = [0, 1, 2]
defines.SEED = 42
defines.AMORTIZED = false
in = 'lfs.c'
code = '''
lfs_t lfs;
lfs_init(&lfs, CFG) => 0;
// create free lookahead
memset(lfs.lookahead.buffer, 0, CFG->lookahead_size);
lfs.lookahead.start = 0;
lfs.lookahead.size = lfs_min(8*CFG->lookahead_size,
CFG->block_count);
lfs.lookahead.next = 0;
lfs_alloc_ack(&lfs);
uint32_t prng = SEED;
// create a tree with N elements
if (AMORTIZED) {
BENCH_START();
}
lfsr_btree_t btree = LFSR_BTREE_NULL;
const char *alphas = "abcdefghijklmnopqrstuvwxyz";
for (lfs_size_t i = 0; i < N; i++) {
lfs_off_t i_
= (ORDER == 0) ? i
: (ORDER == 1) ? 0
: BENCH_PRNG(&prng) % (lfsr_btree_weight(&btree)+1);
lfsr_btree_push(&lfs, &btree, i_, LFSR_TAG_INLINED, 1,
LFSR_DATA_BUF(&alphas[i % 26], 1)) => 0;
}
// bench appending a new id
if (!AMORTIZED) {
BENCH_START();
}
lfs_size_t i = BENCH_PRNG(&prng) % N;
lfsr_btree_push(&lfs, &btree, i, LFSR_TAG_INLINED, 1,
LFSR_DATA_BUF(&alphas[i % 26], 1)) => 0;
BENCH_STOP();
uint8_t buffer[4];
lfsr_tag_t tag_;
lfs_size_t weight_;
lfsr_btree_get(&lfs, &btree, i,
&tag_, &weight_, buffer, 4) => 1;
assert(tag_ == LFSR_TAG_INLINED);
assert(weight_ == 1);
'''