When BENCH_INCLUDE is defined, bench_defines.h should behave like a
normal header file. This includes include guards in case the header file
is included multiple times.
If only for consistency with DISK_GEOMETRY.
The main reason to keep BENCH_PERBYTE around is to help debug/sanity
check the more complex bus+buffer sim. For that purpose it makes sense
to be able to easily switch modes.
The only downside is if it's more difficult to introduce -DDISK_SIM=1 at
runtime vs compile-time, but eh. Consistency wins.
This has been adopted in external benchmarks for a while, as it manages
to push sequential write performance into a much better region of the
diminishing-returns curve.
But hey! Don't take my word for it, let's see the results from our new
bench_runner for the first time:
NOR throughput cs=1/8 cs=1/16
bench_wt_seq+write 15180.0 29402.6 (+93.7%)
bench_wt_random+write 876.2 957.3 (+9.3%)
bench_wt_logging+write 2001.0 2153.4 (+7.6%)
bench_wt_many+write 453.6 453.6 (+0.0%)
NAND throughput cs=1/8 cs=1/16
bench_wt_seq+write 21778.7 22330.1 (+2.5%)
bench_wt_random+write 3583.0 3637.2 (+1.5%)
bench_wt_logging+write 10855.1 10977.1 (+1.1%)
bench_wt_many+write 68.2 68.2 (+0.0%)
Though this doesn't really capture the tradeoffs related to file tails,
storage usage, etc.
In theory sequential writes are happy to start crystallizing as soon as
any data is written, but this leads to significant waste anytime you're
not going to write most of a block.
This gives us much more room for activities.
It makes sense to keep the test disk small: easier parallelization,
heavier emubd with more test features, and if you're running into space
issues in a test, that usually just means you need to be more creative
with how the test is setup.
But for benches, we're interested what happens when we throw a ton of
data at the system.
Also defaulted to noop erases. 0xff erases behave more predictably,
which is useful for testing. But for benching, less work is faster.
This shows an interesting strategy difference between the test_runner
and bench_runner.
In the test_runner we default to the least-stress configuration, to
minimize bugs unrelated to the current test. But the resulting
configuration is unrealistic, as most use cases on flash will probably
want wear-leveling.
In the bench_runner, we should use a more realistic configuration, so
setting BLOCK_RECYCLES=100 by default makes sense.
So now you can easily run multiple/specific geometries without
recompiling the bench runner:
./scripts/bench.py -DDISK_GEOMETRY=0,1
But note by default we only simulate NOR flash. Spitting out multiple
results by default is confusing.
---
Previously this was possible by either compiling multiple bench runners
(with -DBENCH_NAND), or by explicit specifying full the geometry
(-DREAD_SIZE, -DPROG_SIZE, ..., -DREAD_TIMING, ...) at runtime, but both
were clunky and annoying to parameterize.
DISK_GEOMETRY make it easy, fits well with DISK_SIZE, and adds a field
to help identify the geometry in later scripts.
I considered filling out test_defines.h with multiple geometries as
well, but decided against it. The current idea behind test_runner is to
not test specific geometries, but to instead let individual suites/cases
iterate through the specific READ_SIZEs, PROG_SIZEs, etc, that are
relevant. Still, added DISK_GEOMETRY to test_defines.h for consistency,
but it doesn't actually control anything.
After letting it sit for a bit, the previous byte+op sim comes across as
overly clever in a way that is counter-productive. This is highlighted
by erase-timing scaling in a confusing way when per-op.
Fortunately, with a bit of tweaking, we can instead model the bd sim as
separate bus+buffer timings. This seems more intuitive and is closer to
how the actual hardware works.
---
In the bus+buffer model, bd operations are simulated using two sets of
timing estimates:
buffer timings (nor) bus timings (nor)
read_timing (0) readed_timing (40 ns/B)
prog_timing (1563 ns/B) progged_timing (19 ns/B)
erase_timing (10986 ns/B) erased_timing (0)
Bus timings are a simple multiplier of the bytes read/progged/erased,
while buffer timings are rounded up + aligned to the nearest "width":
bd geometry (nor) bd buffers (nor)
read_size (1 B) read_width (1 B)
prog_size (1 B) prog_width (256 B)
erase_size (4096 B) erase_width (4096 B)
For most purposes, the width should just be the device's read/prog/erase
buffer, but I went with the name width to try to keep it generic and
avoid confusion with "buffer" elsewhere in the codebase.
Some notes:
- Like the byte+op sim, the bus+buffer sim allows penalizing small
operations without artificially limiting what operations are possible.
- Because buffer timings depend on read/prog/erase alignment, there's no
simple equation from ops+bytes to bus+buffer. But as a tradeoff, this
new sim more accurately penalizes unaligned operations.
- All timings are still kept as per-byte instead of per-width. This has
proven to be more flexible when benchmarking, as you usually what
timings to scale with the relevant operation.
- Currently this implemented by changing reads/progs/erases to track the
number of "widths" read/progged/erased after alignment. Which makes
the simtime formula roughly:
simtime = reads*read_width*read_timing + readed*readed_timing
(per-butter) (per-bus)
I considered keeping separate counters for calls (read_calls/
prog_calls/erase_calls?), but not sure there's a good reason to. The
theory behind these widths is there no functional difference between
one big call vs multiple width sized calls, though maybe they would be
useful for debugging?
We can always add these later if they turn out to be useful.
- When widths are disable (0), reads/progs/erases reverts to the number
of read/prog/erase calls.
This is the behavior when BENCH_SIMPLE is defined at compile-time.
Having BENCH/TEST_NAND ifdefs that enable the relevant timings, but
_not_ the relevant geometry, is certainly a choice.
Defaulting to NAND geometry when BENCH/TEST_NAND is defined is more
useful, if only for minimizing confusion.
No idea how this ended up with the wrong url! I only noticed when tSE
didn't match what was in the datasheet (expected 45ms, found 50ms).
Ugh. I've been copying this url around for a while now without noticing,
so this is not the only repo that needs fixing...
Initial results with the new timing calculations looked weird. Turns
out different block sizes perform surprisingly when they all cost the
same!
Fortunately, erases are the one operation where per-byte vs per-op
timing doesn't really matter, so reverting to only per-byte timing
solves this problem. Now, erasing 2 4KiB blocks should take the same
time as 1 8KiB block, instead of twice as long.
---
Arguably, erase timing shouldn't be _strictly_ linear w.r.t. block size.
There's a reason denser storage usually ends up with larger block sizes
after all. But preventing the block size from messing with per-byte
timings is much more interesting from a filesystem design perspective.
It also matches the behavior of artificially increasing block size to
reduce block allocator pressure.
Unfortunately, this also raises concerns with read/prog timing when
varying geometry is involved... Should we stick to the per-byte timing
in such cases? Is there a better timing model out there without too much
additional complexity?
The main changes:
- Added TEST_DEFINES and BENCH_DEFINES to allow overriding the default
test/bench define header:
-DTEST_DEFINES=my_test_defines.h
Note these are VERY different from LFS_DEFINES upstream. They aren't a
typical header file, and are included multiple times with various
query macros.
It's hacky, but works surprisingly well.
Or maybe I'll just do anything to avoid having to write multiline
macros. Ugh, backslashes.
- Moved more logic into bench/test_defines.h, including everything
needed to integrate other filesystems out-of-tree.
This mostly meant moving all of the cfg initialization logic into its
own query macro (replacing the BENCH/TEST_CFG field macros).
But this also includes a bunch of small tweaks encountered while trying
to get external benchmarks running again.
The external benchmarks include several other filesystems (littlefs2,
SPIFFS, Yaffs2), and I'm hoping this injectable/queryable header thing
will do a good job at avoiding a maintenance headache. (At least a
better job than forking bench_runner.c, which was the previous
solution.)
kiwibd has been used extensively in external benchmarks, it makes sense
to make it the default bd for the bench runner:
- test_runner - defaults to emubd - more testing features
- bench_runner - defaults to kiwibd - lighter-weight disks
The benefit of kiwibd is the disk is just one big blob of RAM, so
basically no overhead. This is important when benchmarking on multi-GiB
disks.
emubd is much heavy, but as a tradeoff can do quite a bit more:
bad-block simulation, wear simulation, snapshotting, etc.
---
In theory the bd used by each runner can be controlled at compile-time
by defining -DBENCH_EMUBD, etc, but I have a feeling no one will ever
use this.
This is based on some work in external benchmarks. What's worked well
there is emulating a global simtime based on per-byte estimates.
This moves the emulated simtime into emubd/kiwibd, and extends the idea
with both per-byte and per-op timing estimates for hopefully more
realistic results.
---
The problem is how NAND flash reads work.
Per-byte timing estimates are surprisingly accurate for NOR flash. There
is some overhead for sending the address, but it's mostly dominated by
bus cost (~20ns/B [1]).
NAND flash, on the otherhand, technically does support byte-level reads,
but first needs to read into 2KiB buffer. Surprisingly, these are pretty
close in cost (~19ns/B bus [2] vs ~12ns/B buffer [2]).
This close-ness makes modeling NAND flash difficult. If we set
read_size=1, we risk hiding the cost of small reads, which littlefs3 is
full of (rbyd lookups). If we set read_size=2048, we unfairly penalize
littlefs3 for the same reason.
---
The solution here is to expose both per-byte and per-op timing
estimates. This lets you model NAND reads using two data points:
^
| realtime --> ...............o
| : .....'''' :
| ...............:'''' ^ :
| :....''''' | :
| ..........:::::: simtime :
| .....:'''' :
|o....:::::.....: :
|: :
|: :
+:-----------------------------------------------------------:>
min read max read
Where:
bus_timing = 19ns
buffer_timing = 25us
buffer_size = 2KiB
erase_size = 128KiB
min_read = buffer_timing
max_read = (erase_size/buffer_size)*buffer_timing - buffer_timing
read_timing = min_read
readed_timing = ((max_read - min_read)/erase_size) + bus_timing
simtime = reads*read_timing + readed*readed_timing
(per-op) (per-byte)
This should correctly penalize small reads without complicating
emubd/kiwibd too much.
That's the idea anyways! It will take some use to understand if this is
a reasonable approach.
As a plus, this is a superset of the per-byte model, so both can be used
for realistic vs idealistic simulations (and to test the bus+buffer
model itself).
1: https://www.winbond.com/resource-files/W25Q256JV%20SPI%20RevQ%2002072025%20Plus.pdf
2: https://www.winbond.com/resource-files/W25N01GV%20Rev%20R%20070323.pdf
The big TEST_IMPLICIT_DEFINES and TEST_CFG macros have been a big
pain-in-the-ass to maintain. Mostly due to C preprocessor annoyances
(bleh escaped newlines) and no-ifdef workarounds, which make a real mess
of things.
This does two things:
1. Moves all the defines out of test_runner.h and into test_defines.h
(same for benches).
2. Inverts the include logic such that test_defines.h gets included many
times with various "query macros" defined.
Currently just two, but can easily add more:
1. TEST_DEFINE(name, value) - name and default value for a define
2. TEST_CFG(name, value) - name and value for a cfg field
This seems to work surprisingly well. It solves all of the above C
preprocessor issues, and provides a flexible method for defining test
defines.
Note an important part of making this work is that test_defines.h
expands to an empty string by default.
Christopher Haster