After letting it sit for a bit, the previous byte+op sim comes across as
overly clever in a way that is counter-productive. This is highlighted
by erase-timing scaling in a confusing way when per-op.
Fortunately, with a bit of tweaking, we can instead model the bd sim as
separate bus+buffer timings. This seems more intuitive and is closer to
how the actual hardware works.
---
In the bus+buffer model, bd operations are simulated using two sets of
timing estimates:
buffer timings (nor) bus timings (nor)
read_timing (0) readed_timing (40 ns/B)
prog_timing (1563 ns/B) progged_timing (19 ns/B)
erase_timing (10986 ns/B) erased_timing (0)
Bus timings are a simple multiplier of the bytes read/progged/erased,
while buffer timings are rounded up + aligned to the nearest "width":
bd geometry (nor) bd buffers (nor)
read_size (1 B) read_width (1 B)
prog_size (1 B) prog_width (256 B)
erase_size (4096 B) erase_width (4096 B)
For most purposes, the width should just be the device's read/prog/erase
buffer, but I went with the name width to try to keep it generic and
avoid confusion with "buffer" elsewhere in the codebase.
Some notes:
- Like the byte+op sim, the bus+buffer sim allows penalizing small
operations without artificially limiting what operations are possible.
- Because buffer timings depend on read/prog/erase alignment, there's no
simple equation from ops+bytes to bus+buffer. But as a tradeoff, this
new sim more accurately penalizes unaligned operations.
- All timings are still kept as per-byte instead of per-width. This has
proven to be more flexible when benchmarking, as you usually what
timings to scale with the relevant operation.
- Currently this implemented by changing reads/progs/erases to track the
number of "widths" read/progged/erased after alignment. Which makes
the simtime formula roughly:
simtime = reads*read_width*read_timing + readed*readed_timing
(per-butter) (per-bus)
I considered keeping separate counters for calls (read_calls/
prog_calls/erase_calls?), but not sure there's a good reason to. The
theory behind these widths is there no functional difference between
one big call vs multiple width sized calls, though maybe they would be
useful for debugging?
We can always add these later if they turn out to be useful.
- When widths are disable (0), reads/progs/erases reverts to the number
of read/prog/erase calls.
This is the behavior when BENCH_SIMPLE is defined at compile-time.
Having BENCH/TEST_NAND ifdefs that enable the relevant timings, but
_not_ the relevant geometry, is certainly a choice.
Defaulting to NAND geometry when BENCH/TEST_NAND is defined is more
useful, if only for minimizing confusion.
No idea how this ended up with the wrong url! I only noticed when tSE
didn't match what was in the datasheet (expected 45ms, found 50ms).
Ugh. I've been copying this url around for a while now without noticing,
so this is not the only repo that needs fixing...
Initial results with the new timing calculations looked weird. Turns
out different block sizes perform surprisingly when they all cost the
same!
Fortunately, erases are the one operation where per-byte vs per-op
timing doesn't really matter, so reverting to only per-byte timing
solves this problem. Now, erasing 2 4KiB blocks should take the same
time as 1 8KiB block, instead of twice as long.
---
Arguably, erase timing shouldn't be _strictly_ linear w.r.t. block size.
There's a reason denser storage usually ends up with larger block sizes
after all. But preventing the block size from messing with per-byte
timings is much more interesting from a filesystem design perspective.
It also matches the behavior of artificially increasing block size to
reduce block allocator pressure.
Unfortunately, this also raises concerns with read/prog timing when
varying geometry is involved... Should we stick to the per-byte timing
in such cases? Is there a better timing model out there without too much
additional complexity?
The main changes:
- Added TEST_DEFINES and BENCH_DEFINES to allow overriding the default
test/bench define header:
-DTEST_DEFINES=my_test_defines.h
Note these are VERY different from LFS_DEFINES upstream. They aren't a
typical header file, and are included multiple times with various
query macros.
It's hacky, but works surprisingly well.
Or maybe I'll just do anything to avoid having to write multiline
macros. Ugh, backslashes.
- Moved more logic into bench/test_defines.h, including everything
needed to integrate other filesystems out-of-tree.
This mostly meant moving all of the cfg initialization logic into its
own query macro (replacing the BENCH/TEST_CFG field macros).
But this also includes a bunch of small tweaks encountered while trying
to get external benchmarks running again.
The external benchmarks include several other filesystems (littlefs2,
SPIFFS, Yaffs2), and I'm hoping this injectable/queryable header thing
will do a good job at avoiding a maintenance headache. (At least a
better job than forking bench_runner.c, which was the previous
solution.)
kiwibd has been used extensively in external benchmarks, it makes sense
to make it the default bd for the bench runner:
- test_runner - defaults to emubd - more testing features
- bench_runner - defaults to kiwibd - lighter-weight disks
The benefit of kiwibd is the disk is just one big blob of RAM, so
basically no overhead. This is important when benchmarking on multi-GiB
disks.
emubd is much heavy, but as a tradeoff can do quite a bit more:
bad-block simulation, wear simulation, snapshotting, etc.
---
In theory the bd used by each runner can be controlled at compile-time
by defining -DBENCH_EMUBD, etc, but I have a feeling no one will ever
use this.
This is based on some work in external benchmarks. What's worked well
there is emulating a global simtime based on per-byte estimates.
This moves the emulated simtime into emubd/kiwibd, and extends the idea
with both per-byte and per-op timing estimates for hopefully more
realistic results.
---
The problem is how NAND flash reads work.
Per-byte timing estimates are surprisingly accurate for NOR flash. There
is some overhead for sending the address, but it's mostly dominated by
bus cost (~20ns/B [1]).
NAND flash, on the otherhand, technically does support byte-level reads,
but first needs to read into 2KiB buffer. Surprisingly, these are pretty
close in cost (~19ns/B bus [2] vs ~12ns/B buffer [2]).
This close-ness makes modeling NAND flash difficult. If we set
read_size=1, we risk hiding the cost of small reads, which littlefs3 is
full of (rbyd lookups). If we set read_size=2048, we unfairly penalize
littlefs3 for the same reason.
---
The solution here is to expose both per-byte and per-op timing
estimates. This lets you model NAND reads using two data points:
^
| realtime --> ...............o
| : .....'''' :
| ...............:'''' ^ :
| :....''''' | :
| ..........:::::: simtime :
| .....:'''' :
|o....:::::.....: :
|: :
|: :
+:-----------------------------------------------------------:>
min read max read
Where:
bus_timing = 19ns
buffer_timing = 25us
buffer_size = 2KiB
erase_size = 128KiB
min_read = buffer_timing
max_read = (erase_size/buffer_size)*buffer_timing - buffer_timing
read_timing = min_read
readed_timing = ((max_read - min_read)/erase_size) + bus_timing
simtime = reads*read_timing + readed*readed_timing
(per-op) (per-byte)
This should correctly penalize small reads without complicating
emubd/kiwibd too much.
That's the idea anyways! It will take some use to understand if this is
a reasonable approach.
As a plus, this is a superset of the per-byte model, so both can be used
for realistic vs idealistic simulations (and to test the bus+buffer
model itself).
1: https://www.winbond.com/resource-files/W25Q256JV%20SPI%20RevQ%2002072025%20Plus.pdf
2: https://www.winbond.com/resource-files/W25N01GV%20Rev%20R%20070323.pdf
The big TEST_IMPLICIT_DEFINES and TEST_CFG macros have been a big
pain-in-the-ass to maintain. Mostly due to C preprocessor annoyances
(bleh escaped newlines) and no-ifdef workarounds, which make a real mess
of things.
This does two things:
1. Moves all the defines out of test_runner.h and into test_defines.h
(same for benches).
2. Inverts the include logic such that test_defines.h gets included many
times with various "query macros" defined.
Currently just two, but can easily add more:
1. TEST_DEFINE(name, value) - name and default value for a define
2. TEST_CFG(name, value) - name and value for a cfg field
This seems to work surprisingly well. It solves all of the above C
preprocessor issues, and provides a flexible method for defining test
defines.
Note an important part of making this work is that test_defines.h
expands to an empty string by default.
Christopher Haster