The litmus benches are really only intended for introspection/debugging/
cool plots/etc. They're interesting to poke around with and cover a wide
range of littlefs's data-structures, but are not very rigorous.
To make this more clear for new users, added a new litmus flag for
benches:
litmus = true
This doesn't change anything about how the bench is run, but serves as a
marker to hint that the bench is intended for non-rigorous benchmarking.
---
In the makefile, litmus tests are disabled by default at runtime
(--no-litmus). This is to limit `make bench` to benches that are useful
for performance comparisons.
With --no-litmus at runtime, the litmus benches are at least compiled
into the bench_runner, which should hopefully encourage keeping them up
to date with code changes. Eventually we should also run them in CI, but
only to check for runtime errors.
Unlike our tests, we're not really worried about compile time at the
moment due to how few/small our benches are.
Based on some experience out-of-tree:
- bench_rbyd - Simple rbyd attr/id litmus benchmark
- bench_btree (new) - Simple btree id/name litmus benchmark
- bench_file (new) - Simple file read/write litmus benchmark
- bench_dir (new) - Simple dir read/write/stat litmus benchmark
- bench_wt - Heavy-duty write-throughput benchmark
- bench_rt (new) - Heavy-duty read-throughput benchmark
Benches take a long time to run for useful results, so we probably don't
want to go crazy with them like with the tests.
Honestly, we may want to chop this down to just the
write/read-throughput benches.
This is based on how bench.py/bench_runners have actually been used in
practice. The main changes have been to make the output of bench.py more
readibly consumable by plot.py/plotmpl.py without needing a bunch of
hacky intermediary scripts.
Now instead of a single per-bench BENCH_START/BENCH_STOP, benches can
have multiple named BENCH_START/BENCH_STOP invocations to measure
multiple things in one run:
BENCH_START("fetch", i, STEP);
lfsr_rbyd_fetch(&lfs, &rbyd_, rbyd.block, CFG->block_size) => 0;
BENCH_STOP("fetch");
Benches can also now report explicit results, for non-io measurements:
BENCH_RESULT("usage", i, STEP, rbyd.eoff);
The extra iter/size parameters to BENCH_START/BENCH_RESULT also allow
some extra information to be calculated post-bench. This infomation gets
tagged with an extra bench_agg field to help organize results in
plot.py/plotmpl.py:
- bench_meas=<meas>+amor, bench_agg=raw - amortized results
- bench_meas=<meas>+div, bench_agg=raw - per-byte results
- bench_meas=<meas>+avg, bench_agg=avg - average over BENCH_SEED
- bench_meas=<meas>+min, bench_agg=min - minimum over BENCH_SEED
- bench_meas=<meas>+max, bench_agg=max - maximum over BENCH_SEED
---
Also removed all bench.tomls for now. This may seem counterproductive in
a commit to improve benchmarking, but I'm not sure there's actual value
to keeping bench cases committed in tree.
These were alway quick to fall out of date (at the time of this commit
most of the low-level bench.tomls, rbyd, btree, etc, no longer
compiled), and most benchmarks were one-off collections of scripts/data
with results too large/cumbersome to commit and keep updated in tree.
I think the better way to approach benchmarking is a seperate repo
(multiple repos?) with all related scripts/state/code and results
committed into a hopefully reproducible snapshot. Keeping the
bench.tomls in that repo makes more sense in this model.
There may be some value to having benchmarks in CI in the future, but
for that to make sense they would need to actually fail on performance
regression. How to do that isn't so clear. Anyways we can always address
this in the future rather than now.
The previous system of relying on test name prefixes for ordering was
simple, but organizing tests by dependencies and topologically sorting
during compilation is 1. more flexible and 2. simplifies test names,
which get typed a lot.
Note these are not "hard" dependencies, each test suite should work fine
in isolation. These "after" dependencies just hint an ordering when all
tests are ran.
As such, it's worth noting the tests should NOT error of a dependency is
missing. This unfortunately makes it a bit hard to catch typos, but
allows faster compilation of a subset of tests.
---
To make this work the way tests are linked has changed from using custom
linker section (fun linker magic!) to a weakly linked array appended to
every source file (also fun linker magic!).
At least with this method test.py has strict control over the test
ordering, and doesn't depend on 1. the order in which the linker merges
sections, and 2. the order tests are passed to test.py. I didn't realize
the previous system was so fragile.
It doesn't make sense to test more complex logic, such as t2_btree.toml,
when the logic it is built on, t1_rbyd.toml, does not past testing. The
test runner already guarantees a consistent lexicographic order, so all
we need to do is renamed these from test_* -> tn_*.
Note, if we every have more than 10 tests, we will need to bump up the
number of digits for all tests, so t1_rbyd.toml -> t01_rbyd.toml. This
is the main downside of lexicographic ordering. But we'll cross that
bridge when we get to it.
Validating btree nodes during lfsr_btree_lookup was useful as a
proof-of-concept, but it's not really needed if we validate btree nodes
during mtree traversal.
mtree traversal provides the first reads into the filesystem. It's how
we find the real mroot, and (in theory at the moment) it provides the core
operation for error detection in correction. With this in mind,
implementing btree node validation in mtree traversal makes a lot of
sense, with lfsr_btree_lookup leveraging an assumed successful
validation for faster/smaller btree walks.
Note that btree node validation during traversal is still optional. We
really don't want to pay this cost during block allocation for example.
---
It may look concerning that there's no related validation in btree traversal
layer itself.
It turns out that a quirk of btree traversal returning inner btree nodes on
first visit, before actually traversing the btree node, is that it's
safe for us to validte the btree node in only the mtree traversal layer.
As long as we don't continue traversing on finding a corrupted btree,
the btree traversal layer will never traverse an unvalidated btree node.
This keeps all the validation logic in the same place, mtree traversal.
I don't know if this will stay this way if/when more error correction
features are added, but it's convenient in the meantime.
These benchmarks are now more useful for seeing how these B-trees perform.
In plot.py/plotmpl.py:
- Added --legend as another alias for -l, --legend-right.
- Allowed omitting of datasets from the legend by using empty strings
in --labels.
- Do not sum multiple data points on the same x coordinate. This was a
bad idea that risks invalid results going unnoticed.
As a plus multiple data points on the same x coordinate can be abused for
a cheap representation of measurement error.
Christopher Haster