- BENCH_SIMTIME() => lfs3_kiwibd_simtime()
- BENCH_SIMRESET() => lfs3_kiwibd_simreset()
- BENCH_SIMPAUSE() => lfs3_kiwibd_simpause()
- BENCH_SIMRESUME() => lfs3_kiwibd_simresume()
- BENCH_RESET() => lfs3_kiwibd_simreset() + BENCH_STACK/HEAP_RESET()
- BENCH_PAUSE() => lfs3_kiwibd_simpause() + BENCH_STACK/HEAP_PAUSE()
- BENCH_RESUME() => lfs3_kiwibd_simresume() + BENCH_STACK/HEAP_RESUME()
This does two things:
1. Adds pause/resume counters to bd counters to make it easier to
exclude operations from the current bench (potentially useful for
seq+disk usage).
2. Exposes bd simtime operations as BENCH_* macros, to make it a bit
easier to interact with simtime without tying all the benches to
kiwibd. (Not that we'll ever probably not use kiwibd, but still).
Also adopted 32-bit counters for stack/heap pause state instead of a
32-bit stack. Not that either are at a risk of overflowing, but better
safe than sorry.
The litmus benches are really only intended for introspection/debugging/
cool plots/etc. They're interesting to poke around with and cover a wide
range of littlefs's data-structures, but are not very rigorous.
To make this more clear for new users, added a new litmus flag for
benches:
litmus = true
This doesn't change anything about how the bench is run, but serves as a
marker to hint that the bench is intended for non-rigorous benchmarking.
---
In the makefile, litmus tests are disabled by default at runtime
(--no-litmus). This is to limit `make bench` to benches that are useful
for performance comparisons.
With --no-litmus at runtime, the litmus benches are at least compiled
into the bench_runner, which should hopefully encourage keeping them up
to date with code changes. Eventually we should also run them in CI, but
only to check for runtime errors.
Unlike our tests, we're not really worried about compile time at the
moment due to how few/small our benches are.
Based on some experience out-of-tree:
- bench_rbyd - Simple rbyd attr/id litmus benchmark
- bench_btree (new) - Simple btree id/name litmus benchmark
- bench_file (new) - Simple file read/write litmus benchmark
- bench_dir (new) - Simple dir read/write/stat litmus benchmark
- bench_wt - Heavy-duty write-throughput benchmark
- bench_rt (new) - Heavy-duty read-throughput benchmark
Benches take a long time to run for useful results, so we probably don't
want to go crazy with them like with the tests.
Honestly, we may want to chop this down to just the
write/read-throughput benches.
- Renamed BENCH_STACK/HEAP -> BENCH_STACK/HEAP_WATERMARK
- Renamed BENCH_YES_STACK/HEAP -> BENCH_STACK/HEAP
- Tweaked stack/heap watermarks to hopefully be easier to access when
debugging. Now also exposed as global variables
(bench_stack/heap_watermark).
I considered changing BENCH_STACK/HEAP_WATERMARK to be the variable
itself, to be consistent with TEST_PLS, but decided against it:
1. BENCH_STACK_CURRENT() is a bit magic in that it relies on
__attribute__((noinline)) to force a new stack frame. This wouldn't
really be possible with a variable.
2. TEST_PLS is at least constant from the _current run_'s perspective.
This isn't true for the stack/heap watermarks.
- Reworked internals a bit to hopefully be simpler
Note bench_wt_seq's disk usage is garbage because of the repeated
truncates!
But I figured this is at least useful for the other benches, and we
already have bench_helper_usage. Maybe in the future we'll figure out
some way to get useful disk usage from bench_wt_seq.
Might as well, we have the hooks already.
The only annoying this is these extra probes clutter up the `make
bench-marks` output, so split into two separate rules:
- make bench-marks
- make bench-usage
It's tempting to add disk usage as well (we have bench_helper_usage for
this purpose), but I'm not sure how to measure usage in bench_wt_seq
since it's constantly truncating.
This gives us much more room for activities.
It makes sense to keep the test disk small: easier parallelization,
heavier emubd with more test features, and if you're running into space
issues in a test, that usually just means you need to be more creative
with how the test is setup.
But for benches, we're interested what happens when we throw a ton of
data at the system.
Also defaulted to noop erases. 0xff erases behave more predictably,
which is useful for testing. But for benching, less work is faster.
These were copied from external benchmarks, and tweaked/simplified a
bit based on gained experience.
I mostly just wanted something to test the bench runner/scripts, with
bench_rbyd showcasing a low-level litmus benchmark, and bench_wt
showcasing a high-level throughput benchmark.
Though bench_wt has proven to be a _very_ versatile benchmark, and will
likely be the first stop for getting an understanding of high-level
performance implications.
---
Also added bench_helpers.h/c, which includes a couple helper functions:
- bench_helpers_warmup - Warm up the filesystem by writing a 1 block
file 2*block_count times. This is meant to exhaust any preerased
state, post-format lookahead buffers, etc.
- bench_helpers_usage - Find a tight bound on disk usage. This allocates
a bitmap to find the tight bound, unlike lfs3_fs_usage, which is
best-effort. However the bitmap is hidden behind BENCH_HEAP_PAUSE to
prevent messing with parallel heap measurements.
This is based on how bench.py/bench_runners have actually been used in
practice. The main changes have been to make the output of bench.py more
readibly consumable by plot.py/plotmpl.py without needing a bunch of
hacky intermediary scripts.
Now instead of a single per-bench BENCH_START/BENCH_STOP, benches can
have multiple named BENCH_START/BENCH_STOP invocations to measure
multiple things in one run:
BENCH_START("fetch", i, STEP);
lfsr_rbyd_fetch(&lfs, &rbyd_, rbyd.block, CFG->block_size) => 0;
BENCH_STOP("fetch");
Benches can also now report explicit results, for non-io measurements:
BENCH_RESULT("usage", i, STEP, rbyd.eoff);
The extra iter/size parameters to BENCH_START/BENCH_RESULT also allow
some extra information to be calculated post-bench. This infomation gets
tagged with an extra bench_agg field to help organize results in
plot.py/plotmpl.py:
- bench_meas=<meas>+amor, bench_agg=raw - amortized results
- bench_meas=<meas>+div, bench_agg=raw - per-byte results
- bench_meas=<meas>+avg, bench_agg=avg - average over BENCH_SEED
- bench_meas=<meas>+min, bench_agg=min - minimum over BENCH_SEED
- bench_meas=<meas>+max, bench_agg=max - maximum over BENCH_SEED
---
Also removed all bench.tomls for now. This may seem counterproductive in
a commit to improve benchmarking, but I'm not sure there's actual value
to keeping bench cases committed in tree.
These were alway quick to fall out of date (at the time of this commit
most of the low-level bench.tomls, rbyd, btree, etc, no longer
compiled), and most benchmarks were one-off collections of scripts/data
with results too large/cumbersome to commit and keep updated in tree.
I think the better way to approach benchmarking is a seperate repo
(multiple repos?) with all related scripts/state/code and results
committed into a hopefully reproducible snapshot. Keeping the
bench.tomls in that repo makes more sense in this model.
There may be some value to having benchmarks in CI in the future, but
for that to make sense they would need to actually fail on performance
regression. How to do that isn't so clear. Anyways we can always address
this in the future rather than now.
The previous system of relying on test name prefixes for ordering was
simple, but organizing tests by dependencies and topologically sorting
during compilation is 1. more flexible and 2. simplifies test names,
which get typed a lot.
Note these are not "hard" dependencies, each test suite should work fine
in isolation. These "after" dependencies just hint an ordering when all
tests are ran.
As such, it's worth noting the tests should NOT error of a dependency is
missing. This unfortunately makes it a bit hard to catch typos, but
allows faster compilation of a subset of tests.
---
To make this work the way tests are linked has changed from using custom
linker section (fun linker magic!) to a weakly linked array appended to
every source file (also fun linker magic!).
At least with this method test.py has strict control over the test
ordering, and doesn't depend on 1. the order in which the linker merges
sections, and 2. the order tests are passed to test.py. I didn't realize
the previous system was so fragile.
It doesn't make sense to test more complex logic, such as t2_btree.toml,
when the logic it is built on, t1_rbyd.toml, does not past testing. The
test runner already guarantees a consistent lexicographic order, so all
we need to do is renamed these from test_* -> tn_*.
Note, if we every have more than 10 tests, we will need to bump up the
number of digits for all tests, so t1_rbyd.toml -> t01_rbyd.toml. This
is the main downside of lexicographic ordering. But we'll cross that
bridge when we get to it.
Note that because we amortize the traversal cost over the number of
entries, mtree traversal may have some strange looking results when
compared to mtree lookup.
Though it's interesting to note this is a valid result. In mtree lookups
we need to fetch the mdir for each entry, which is expensive. However
mtree traversal can strictly avoid fetching each mdir more than once.
This does make mdir traversal faster when iterating over all mdirs in
order.
This can be represented in big O notation if we treat the number of
entries (n) and block size (b) as variables:
- mtree traversal via lookup = O(nb+nlog(b)logb(n))
- mtree traversal via traversal = O(nlog(b)logb(n))
Validating btree nodes during lfsr_btree_lookup was useful as a
proof-of-concept, but it's not really needed if we validate btree nodes
during mtree traversal.
mtree traversal provides the first reads into the filesystem. It's how
we find the real mroot, and (in theory at the moment) it provides the core
operation for error detection in correction. With this in mind,
implementing btree node validation in mtree traversal makes a lot of
sense, with lfsr_btree_lookup leveraging an assumed successful
validation for faster/smaller btree walks.
Note that btree node validation during traversal is still optional. We
really don't want to pay this cost during block allocation for example.
---
It may look concerning that there's no related validation in btree traversal
layer itself.
It turns out that a quirk of btree traversal returning inner btree nodes on
first visit, before actually traversing the btree node, is that it's
safe for us to validte the btree node in only the mtree traversal layer.
As long as we don't continue traversing on finding a corrupted btree,
the btree traversal layer will never traverse an unvalidated btree node.
This keeps all the validation logic in the same place, mtree traversal.
I don't know if this will stay this way if/when more error correction
features are added, but it's convenient in the meantime.
It's interesting to note the different performance characteristics of
purely CoW btrees vs our mutable mtree.
The main downside of our mtree is the need to fetch leaf mdirs. This
fetch is expensive, and can be avoided in CoW btrees by storing the
trunk in each branch's parent.
On the other hand, btrees need to propagate all changes upwards to the
root.
An interesting takeaway is that a sort of mdir-trunk cache may be a very
interesting optimization for relatively little RAM cost. This may be
something to explore in the future.
These benchmarks are now more useful for seeing how these B-trees perform.
In plot.py/plotmpl.py:
- Added --legend as another alias for -l, --legend-right.
- Allowed omitting of datasets from the legend by using empty strings
in --labels.
- Do not sum multiple data points on the same x coordinate. This was a
bad idea that risks invalid results going unnoticed.
As a plus multiple data points on the same x coordinate can be abused for
a cheap representation of measurement error.
- Added both uattr (limited to 256) and id (limited to 65535) benchmarks
covering the main rbyd operations
- Fixed issue where --defines gets passed to the test/bench runners when
querying id-specific information. After changing the test/bench
runners to prioritize explicit defines, this causes problems for
recorded benchmark results and debug related things.
- In plot.py/plotmpl.py, made --by/-x/-y in subplots behave somewhat
reasonably, contributing to a global dataset and the figure's legend,
colors, etc, but only shown in the specified subplot. This is useful
mainly for showing different -y values on different subplots.
- In plot.py/plotmpl.py, added --labels to allow explicit configuration
of legend labels, much like --colors/--formats/--chars/etc. This
removes one of the main annoying needs for modifying benchmark results.
When you add a function to every benchmark suite, you know if should
probably be provided by the benchmark runner itself. That being said,
randomness in tests/benchmarks is a bit tricky because it needs to be
strictly controlled and reproducible.
No global state is used, allowing tests/benches to maintain multiple
randomness stream which can be useful for checking results during a run.
There's an argument for having global prng state in that the prng could
be preserved across power-loss, but I have yet to see a use for this,
and it would add a significant requirement to any future test/bench runner.
These are really just different flavors of test.py and test_runner.c
without support for power-loss testing, but with support for measuring
the cumulative number of bytes read, programmed, and erased.
Note that the existing define parameterization should work perfectly
fine for running benchmarks across various dimensions:
./scripts/bench.py \
runners/bench_runner \
bench_file_read \
-gnor \
-DSIZE='range(0,131072,1024)'
Also added a couple basic benchmarks as a starting point.
Christopher Haster