Github/littlefs
1
Fork 0
Code Actions Activity

Reworked test/bench defines to map to global variables

Browse Source 
Motivation:

- Debuggability. Accessing the current test/bench defines from inside
  gdb was basically impossible for some dumb macro-debug-info reason I
  can't figure out.

  In theory, GCC provides a .debug_macro section when compiled with -g3.
  I can see this section with objdump --dwarf=macro, but somehow gdb
  can't seem to find any definitions? I'm guess the #line source
  remapping is causing things to break somehow...

  Though even if macro-debugging gets fixed, which would be valuable,
  accessing defines in the current test/bench runner can trigger quite
  a bit of hidden machinery. This risks side-effects, which is never
  great when debugging.

  All of this is quite annoying because the test/bench defines is
  usually the most important piece of information when debugging!

  This replaces the previous hidden define machinery with simple global
  variables, which gdb can access no problem.

- Also when debugging we no longer awkwardly step into the test_define
  function all the time!

- In theory, global variables, being a simple memory access, should be
  quite a bit faster than the hidden define machinery. This does matter
  because running tests _is_ a dev bottleneck.

  In practice though, any performance benefit is below the noise floor,
  which isn't too surprising (~630s +-~20s).

- Using global variables for defines simplifies the test/bench runner
  quite a bit.

  Though some of the previous complexity was due to a whole internal
  define caching system, which was supposed to lazily evaluate test
  defines to avoid evaluating defines we don't use. This all proved to
  be useless because the first thing we do when running each test is
  evaluate all defines to generate the test id (lol).

So now, instead of lazily evaluating and caching defines, we just
generate global variables during compilation and evaluate all defines
for each test permutation immediately before running.

This relies heavily on __attribute__((weak)) symbols, and lets the
linker really shine.

As a funny perk this also effectively interns all test/bench defines by
the address of the resulting global variable. So we don't even need to
do string comparisons when mapping suite-level defines to the
runner-level defines.

---

Perhaps the more interesting thing to note, is the change in strategy in
how we actually evaluate the test defines.

This ends up being a surprisingly tricky problem, due to the potential
of mutual recursion between our defines.

Previously, because our define machinery was lazy, we could just
evaluate each define on demand. If a define required another define, it
would lazily trigger another evaluation, implicitly recursing through
C's stack. If cyclic, this would eventually lead to a stack overflow,
but that's ok because it's a user error to let this happen.

The "correct" way, at least in terms of being computationally optimal,
would be to topologically sort the defines and evaluate the resulting
tree from the leaves up.

But I ain't got time for that, so the solution here is equal parts
hacky, simple, and effective.

Basically, we just evaluate the defines repeatedly until they stop
changing:

- Initially, mutually recursive defines may read the uninitialized
  values of their dependencies, and end up with some arbitrarily wrong
  result. But as the defines are repeatedly evaluated, assuming no
  cycles, the correct results should eventually bubble up the tree until
  all defines converge to the correct value.

- This is O(n*e) vs O(n+e), but our define graph is usually quite
  shallow.

- To prevent non-halting, we error after an arbitrary 1000 iterations.
  If you hit this, it's likely because there is a cycle in the define
  graph.

  This is runtime configurable via the new --define-depth flag.

- To keep things consistent and reproducible, we zero initialize all
  defines before the first evaluation.

  I don't think this is strictly necessary, but it's important for the
  test runner to have the exact same results on every run. No one wants
  a "works on my machine" situation when the tests are involved.

Experimentation shows we only need an evaluation depth of 2 to
successfully evaluate the current set of defines:

  $ ./runners/test_runner --list-defines --define-depth=2

And any performance impact is negligible (~630s +-~20s).
This commit is contained in:
Christopher Haster
2024-02-13 17:21:21 -06:00
parent ddb86af059
commit a124ee54e7
6 changed files with 795 additions and 882 deletions
Download Patch File Download Diff File Expand all files Collapse all files
runners/bench_runner.h
+28 -59
View File
@@ -59,6 +59,8 @@ enum bench_flags {
typedef uint8_t bench_flags_t;
typedef struct bench_define {
const char *name;
intmax_t *define;
intmax_t (*cb)(void *data, size_t i);
void *data;
size_t permutations;
@@ -81,7 +83,7 @@ struct bench_suite {
const char *path;
bench_flags_t flags;
const char *const *define_names;
const bench_define_t *defines;
size_t define_count;
const struct bench_case *cases;
@@ -105,68 +107,35 @@ void bench_permutation(size_t i, uint32_t *buffer, size_t size);
#define BENCH_PERMUTATION(i, buffer, size) bench_permutation(i, buffer, size)
// access generated bench defines
intmax_t bench_define(size_t define);
#define BENCH_DEFINE(i) bench_define(i)
// a few preconfigured defines that control how benches run
#define READ_SIZE_i 0
#define PROG_SIZE_i 1
#define BLOCK_SIZE_i 2
#define BLOCK_COUNT_i 3
#define DISK_SIZE_i 4
#define CACHE_SIZE_i 5
#define INLINE_SIZE_i 6
#define SHRUB_SIZE_i 7
#define FRAGMENT_SIZE_i 8
#define CRYSTAL_THRESH_i 9
#define LOOKAHEAD_SIZE_i 10
#define BLOCK_CYCLES_i 11
#define ERASE_VALUE_i 12
#define ERASE_CYCLES_i 13
#define BADBLOCK_BEHAVIOR_i 14
#define POWERLOSS_BEHAVIOR_i 15
#define BENCH_IMPLICIT_DEFINE_COUNT 16
#define READ_SIZE bench_define(READ_SIZE_i)
#define PROG_SIZE bench_define(PROG_SIZE_i)
#define BLOCK_SIZE bench_define(BLOCK_SIZE_i)
#define BLOCK_COUNT bench_define(BLOCK_COUNT_i)
#define DISK_SIZE bench_define(DISK_SIZE_i)
#define CACHE_SIZE bench_define(CACHE_SIZE_i)
#define INLINE_SIZE bench_define(INLINE_SIZE_i)
#define SHRUB_SIZE bench_define(SHRUB_SIZE_i)
#define FRAGMENT_SIZE bench_define(FRAGMENT_SIZE_i)
#define CRYSTAL_THRESH bench_define(CRYSTAL_THRESH_i)
#define LOOKAHEAD_SIZE bench_define(LOOKAHEAD_SIZE_i)
#define BLOCK_CYCLES bench_define(BLOCK_CYCLES_i)
#define ERASE_VALUE bench_define(ERASE_VALUE_i)
#define ERASE_CYCLES bench_define(ERASE_CYCLES_i)
#define BADBLOCK_BEHAVIOR bench_define(BADBLOCK_BEHAVIOR_i)
#define POWERLOSS_BEHAVIOR bench_define(POWERLOSS_BEHAVIOR_i)
#define BENCH_IMPLICIT_DEFINES \
/* name value (overridable) */ \
BENCH_DEF(READ_SIZE, 1 ) \
BENCH_DEF(PROG_SIZE, 1 ) \
BENCH_DEF(BLOCK_SIZE, 4096 ) \
BENCH_DEF(BLOCK_COUNT, DISK_SIZE/BLOCK_SIZE ) \
BENCH_DEF(DISK_SIZE, 1024*1024 ) \
BENCH_DEF(CACHE_SIZE, lfs_max(16, lfs_max(READ_SIZE, PROG_SIZE))) \
BENCH_DEF(INLINE_SIZE, BLOCK_SIZE/4 ) \
BENCH_DEF(SHRUB_SIZE, INLINE_SIZE ) \
BENCH_DEF(FRAGMENT_SIZE, CACHE_SIZE ) \
BENCH_DEF(CRYSTAL_THRESH, BLOCK_SIZE/8 ) \
BENCH_DEF(LOOKAHEAD_SIZE, 16 ) \
BENCH_DEF(BLOCK_CYCLES, -1 ) \
BENCH_DEF(ERASE_VALUE, 0xff ) \
BENCH_DEF(ERASE_CYCLES, 0 ) \
BENCH_DEF(BADBLOCK_BEHAVIOR, LFS_EMUBD_BADBLOCK_PROGERROR ) \
BENCH_DEF(POWERLOSS_BEHAVIOR, LFS_EMUBD_POWERLOSS_NOOP )
BENCH_DEFINE(READ_SIZE, 1 ) \
BENCH_DEFINE(PROG_SIZE, 1 ) \
BENCH_DEFINE(BLOCK_SIZE, 4096 ) \
BENCH_DEFINE(BLOCK_COUNT, DISK_SIZE/BLOCK_SIZE ) \
BENCH_DEFINE(DISK_SIZE, 1024*1024 ) \
BENCH_DEFINE(CACHE_SIZE, \
lfs_max(16, lfs_max(READ_SIZE, PROG_SIZE))) \
BENCH_DEFINE(INLINE_SIZE, BLOCK_SIZE/4 ) \
BENCH_DEFINE(SHRUB_SIZE, INLINE_SIZE ) \
BENCH_DEFINE(FRAGMENT_SIZE, CACHE_SIZE ) \
BENCH_DEFINE(CRYSTAL_THRESH, BLOCK_SIZE/8 ) \
BENCH_DEFINE(LOOKAHEAD_SIZE, 16 ) \
BENCH_DEFINE(BLOCK_CYCLES, -1 ) \
BENCH_DEFINE(ERASE_VALUE, 0xff ) \
BENCH_DEFINE(ERASE_CYCLES, 0 ) \
BENCH_DEFINE(BADBLOCK_BEHAVIOR, LFS_EMUBD_BADBLOCK_PROGERROR ) \
BENCH_DEFINE(POWERLOSS_BEHAVIOR, LFS_EMUBD_POWERLOSS_NOOP )
// declare defines as global intmax_ts
#define BENCH_DEFINE(k, v) \
extern intmax_t k;
BENCH_IMPLICIT_DEFINES
#undef BENCH_DEFINE
// map defines to cfg struct fields
#define BENCH_CFG \
.read_size = READ_SIZE, \
.prog_size = PROG_SIZE, \