Looks like this was never updated after changing the -Y/--summary +
-c/--compare hack to its own -Q/--small-table flag. Fortunately a single
character fix.
Unrelated, but I was considering dropping the make *-diff rules, until
the different compile-time targets proved they are _very_ useful when
jumping around various commits/builds.
This reverts per-result source file mapping, and tears out of a bunch of
messy dwarf parsing code. Results from the same .o file are now mapped
to the same source file.
This was just way too much complexity for slightly better result->file
mapping, which risked losing results accidentally mapped to the wrong
file.
---
I was originally going to revert all the way back to relying strictly on
the .o name and --build-dir (490e1c4) (this is the simplest solution),
but after poking around in dwarf-info a bit, I realized we do have
access to the original source file in DW_TAG_compile_unit's
DW_AT_comp_dir + DW_AT_name.
This is much simpler/more robust than parsing objdump --dwarf=rawline,
and avoid needing --build-dir in a bunch of scripts.
---
This also reverts stack.py to rely only on the .ci files. These seem as
reliable as DW_TAG_compile_unit while simplifying things significantly.
Symbol mapping used to be a problem, but this was fixed by using the
symbol in the title field instead of the label field (which strips some
optimization suffixes?)
See previous commit for the issues with stack.py's current approach. I'm
convinced dwarf-info simply does not contain enough info to figure out
stack usage.
There is one last idea, which is to parse the dissassembly. In theory
you only need to understand calls, branches (for control-flow), and
push/pop instructions to figure out the worst-case stack usage. But this
would be ISA-specific and error-prone, so it probably shouldn't
_replace_ the -fcallgraph-info=su based stack.py.
So, out of ideas, reverting.
---
It's worth noting this isn't a trivial revert. There's a couple
interesting changes in stack.py:
- We now use .o files to map callgraph nodes to relevant symbol names.
This should be a bit more robust than relying only on the names in the
.ci files, and guarantees function names line up with other
symbol-based scripts (code.py, ctx.py, etc).
This also lets us warn on missing callgraph nodes, in case the
callgraph info is incomplete.
- Callgraph parsing should be quite a bit more robust now. Added a small
(and reusable?) Parser class.
- Moved cycle detection into result collection.
This should let us drop cycle detection from the table renderer
eventually.
There were a lot of small challenges (see previous commits), but this
commit reworks stack.py to rely only on dwarf-info and symbols to build
stack + callgraph info.
Not only does this remove an annoying dependency on a GCC-specific flag,
but it also should give us more correct stack measurements by only
penalizing calls for the stack usage at the call site. This should
better account for things like shrinkwrapping, which make the
-fcallgraph-info=su results look worse than they actually are.
To make this work required jumping through a couple hoops:
1. Map symbols -> dwarf entries by address (DW_AT_low_pc).
We use symbols here to make sure function names line up with other
scripts.
Note that there can be multiple dwarf entries with the same name due
to optimization passes. Apparently the optimized name is not included
because that would be too useful.
2. Find each functions' frame info.
This is stored in the .debug_frames section (objdump --dwarf=frames),
and requires _yet another state machine_ to parse, but gives us the
stack frame info for each function at the instruction level, so
that's nice.
3. Find call sites (DW_TAG_call_site).
The hierchical nesting of DW_TAG_lexical_blocks gets a bit annoying
here, but ultimately we can find all DW_TAG_call_sites by looking at
the DW_TAG_subprogram's children tags.
4. Map call sites to frame info.
This gets funky.
Finding the target function is simple enough, DW_AT_call_origin
contains its dwarf offset (but why is this the _origin_?). But we
don't actually know what address the call originated from.
Fortunately we do know the return address, DW_AT_call_return_pc?
The instruction before DW_AT_call_return_pc should be the call
instruction. Subtracting 1 will awkwardly put us in the middle of the
instruction, but it should at least map to the correct stack frame?
And without ISA-specific info it's the best we can do.
It's messy, but this should be all the info we need.
---
To build confidence in the new script, I included the --no-shrinkwrap
flag, which reverts to penalizing each call site for the function's
worst-case stack frame. This makes it easy to compare against the
-fcallgraph-info=su approach:
with -fcallgraph-info=su: 2624
with --dwarf=info --no-shrinkwrap: 2624
I was hoping that accounting for shrinkwrap-like optimizations would
reveal a lower stack cost, but for better or worse it seems that
worst-case stack usage is unchanged:
with --dwarf=info --no-shrinkwrap: 2624
with --dwarf=info: 2624
Still, it's good to know that our stack measurement is correct.
There is an argument for prefering nm for code size measurements due to
portability. But I'm not sure this really holds up these days with
objdump being so prevalent.
We already depend on objdump for ctx/structs/perf and other dwarf info,
so using objdump -t to get symbol information means one less tool to
depend on/pass around when cross-compiling.
As a minor benefit this also gives us more control over which sections
to include, instead of relying on nm's predefined t/r/d/b section types.
---
Note code.py/data.py did _not_ require objdump before this. They did use
objdump to map symbols to source files, but would just guess if
objdump wasn't available.
make ctx now does what you expect it to, and ctx.py now replaces
structs.py in the summary rules (make funcs, make summary):
$ make summary
... blablabla ...
code data stack ctx
TOTAL 38100 0 2624 752
Also finally cleaned up SUMMARYFLAGS in make funcs. This should have
been cleaned up when cleaning up make summary...
This showcases the sort of high-level result printing where -c/--compare
is useful:
$ make summary-diff
code data stack structs
BEFORE 57057 0 3056 1476
AFTER 68864 (+20.7%) 0 (+0.0%) 3744 (+22.5%) 1520 (+3.0%)
There was one hiccup though: how to hide the name of the first field.
It may seem minor, but the missing field name really does help
readability when you're staring at a wall of CLI output.
It's a bit of a hack, but this can now be controlled with -Y/--summary,
which has the sole purpose of disabling the first field name if mixed
with -c/--compare.
-c/--compare is already a weird case for the summary row anyways...
I still think the 24 (23+1) char minimum is a good default for 2 column
output such as help text, especially if you don't have automatic width
detection. But our result scripts need to be a bit more flexible.
Consider:
$ make summary
code data stack structs
TOTAL 68864 0 3744 1520
Vs:
$ make summary
code data stack structs
TOTAL 68864 0 3744 1520
Up until now we were just kind of working around this with cut -c 25- in
our Makefile, but now that our result scripts automatically scale the
table widths, they should really just default to whatever is the most
useful.
This seems like a more fitting name now that this script has evolved
into more of a general purpose high-level CSV tool.
Unfortunately this does conflict with the standard csv module in Python,
breaking every script that imports csv (which is most of them).
Fortunately, Python is flexible enough to let us remove the current
directory before imports with a bit of an ugly hack:
# prevent local imports
__import__('sys').path.pop(0)
These scripts are intended to be standalone anyways, so this is probably
a good pattern to adopt.
With the adoption of the odd-parity-zero rbyd perturb scheme, it's now
possible to validate individual tag's parity with neighboring valid
bits. This sparked an idea that I previously thought was intractable.
If we:
1. Validate all metadata reads by checking their on-disk parity bits.
2. Validate all data reads by checking their in-metadata checksums.
We end up with a closed system where all reads are checked by at least
a parity bit.
Being able to check all reads is a very valuable filesystem feature, but
difficult for littlefs:
- We need to keep relevant data in RAM while validating checksums.
We can't just validate checksums and then perform a second read as
that creates a hole where new bit-errors may be introduced.
- This is solved in other filesystems by loading and checking whole
blocks in RAM. We just can't do that here.
- Without parity, we would need to check the rbyd's checksum on every
tag read. This would lead to a crazy O(n^2 log n) rbyd compaction
runtime.
Which is why I original thought ckreads was just intractable.
Now, this isn't all sunshine and rainbows. ckreads, as implemented here,
has some deeply concerning flaws:
- A parity bit is, mathematically, the minimum possible error-detection
possible. Is validating reads with only a parity bit sufficient for
real world applications?
- Validating data checksums on every read may have severe performance
implications. We need to read up to the entire block, which can lead
to O(n^2) behavior when performing a lot of small reads in a file.
- In order to validate checksums/parity-bits, we need to know where the
checksums/parity-bits actually are for each piece of data.
Our lfsr_data_t struct provides a surprisingly nice abstraction for
this, but oof is it expensive.
For the added code/stack cost alone, we probably want to eventually make
this an opt-in compile-time feature.
---
Implementation notes:
- This found an actual compiler bug! Turns out increasing lfsr_data_t
from 3-words to 5-words confuses GCC:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101854
- Mid-commit, we may have not actually written the last tag's parity
yet, which is a bit of a problem because we may read the last tag when
building the next trunk!
Fixing this required a whole separate tailck mechanism, which just
tracks in-progress commit's parity bits.
This doesn't help the code/stack cost situation...
- lfsr_bd_read/cmp/cpy all need to be extended to support calculating a
checksum on the side, which is a bit of a mess.
- bptr's cksize/cksum is redundant now, which is going to make
conditional compilation a mess.
- The extra parity byte we need to read makes hint calculation a pain.
Code cost wise... yeah, it's significant. Turns out almost doubling
lfsr_data_t has a significant impact on stack usage. Add in all the
extra code to track checksums/parity-bits and validate checksums/
parity-bits and you got yourself a pretty heavy feature:
code stack
before: 36352 2672
after: 38100 (+4.8%) 3032 (+13.5%)
The original idea was to allow merging a whole bunch of different csv
results into a single lfs.csv file, but this never really happened. It's
much easier to operate on smaller context-specific csv files, where the
field prefix:
- Doesn't really add much information
- Requires more typing
- Is confusing in how it doesn't match the table field names.
We can always use summary.py -fcode_size=size to add prefixes when
necessary anyways.
Now, fractions are considered equal if they have the same ratio:
- 6/6 == 12/12 => True
- 3/6 == 3/12 => False
- 1/6 == 2/12 => True
It's interesting to note this implementation is actually more
numerically stable than float comparison, though that wasn't really the
goal.
The main reason for this is to allow other fields to take over when
sorting multi-field fractional data: cov (lines + branches), testmarks
(passed + time), etc. Before, sorting would usually stop after
mismatched fraction fields, which wasn't all that useful.
- Renamed build-test -> build-tests
- Renamed build-bench -> build-benches
- Added list-tests alias
- Added list-benches alias
Also made the Makefile's help text generation a bit more robust to long
rule names, which are common in Makefiles. If the name is >=21 chars, we
just indent, similar to test/bench_runner --help.
Note these are different than TESTFLAGS/BENCHFLAGS:
- TEST_CFLAGS/BENCH_CFLAGS => gcc $(TEST_CFLAGS) lfs.t.a.c -o lfs.t.a.o
- TESTFLAGS/BENCHFLAGS => ./scripts/test.py $(TESTFLAGS)
Also tried to group the src/tools/flags a bit better.
These provide useful file powerloss testing that scales linearly as long
as progress can be made. They can still struggle a bit, especially with
relocations which often fail to make progress, but they are _much_ better
than the O(n^2) simulation-based fuzz tests:
- test_files_pl_fuzz - 258734 pls
- test_relocations_pl_fuzz - 928638 pls
Our current problem with simulation-based fuzz testing is that we lose
the simulation on powerloss. We could brute force this, repeatedly
rerunning the simulation until it succeeds, but this grows O(n^2) with
our linear powerloss heuristic.
To avoid this, test_*_pl_fuzz doesn't bother with a simulation, instead
relying on internal asserts to catch bugs. This is less rigorous, but
realistically probably going to catch any powerloss related issues.
Some notes:
- We need to store some state on disk. If we don't we will still end up
with O(n^2) behavior because we simply don't know how many operations
we've accomplished so far.
- Since we rely on file operations to store our test state, this makes
this approach incompatible with the dir tests, which assume file
operations may not yet be implemented.
We still use O(n^2) powerloss testing in test_dirs, just with a small
number of directories.
- It's tempting to try to store a full simulation on disk. But you
would quickly run into atomicity issues with the simulation itself.
Powerloss resilience is tricky!
- We can at least store a checksum in the files (currently just mod 26)
to check that the file itself was not corrupted. This doesn't protect
against swapped data though.
---
Also, a bit of a tangent, but I needed to add -Wno-format-overflow to
the test flags to avoid an annoying invalid format-overlow warning:
struct lfs_info info;
char name[256];
if (strlen(info.name) < 100) { // can't overflow!?
sprintf(name, "test/%s", info.name); // <--
}
warning: '%s' directive writing up to 255 bytes into a region of size
251 [-Wformat-overflow=]
This seems like a GCC bug, because as far as I can tell there is no way
to signal or hint that the size is in bounds without just disabling the
warning completely...
The move to lfs_memcmp/lfs_strcmp highlighted an interesting hole in
prettyasserts.py: the lack of support for custom memcmp/strcmp symbols.
Rather than just adding more flags for an increasing number of symbols,
I've added -p/--prefix and -P/--prefix-insensitive to generate relevant
symbols based on a prefix. In littlefs's case, we use -Plfs_, which
matches both lfs_memcmp and LFS_ASSERT (and LFS_MEMCMP and lfs_assert
but ignore those):
$ ./scripts/prettyasserts.py -Plfs_ lfs.t.c -o lfs.t.a.c
Don't worry, you can still provide explicit symbols, but only via
long-form flags. This gets a bit noisy:
$ ./scripts/prettyasserts.py \
--assert=LFS_ASSERT \
--unreachable=LFS_UNREACHABLE \
--memcmp=lfs_memcmp \
--strcmp=lfs_strcmp \
lfs.t.c -o lfs.t.a.c
This commit also finally gives the prettyasserts.py's symbols actual
word boundaries, instead of the big error-prone hack of sorting by size.
- YES_COV -> COVGEN
- YES_PERF -> PERFGEN
- YES_PERFBD -> PERFBDGEN
- YES_TESTMARKS -> TESTMARKS
- YES_BENCHMARKS -> BENCHMARKS
For a lack of better naming, may change in the future. Couldn't really
find any consistent prior art.
Note that no-suf/prefix doesn't work because of conflicts with the tool
overrides (PERF, COV, etc).
- NO_COV -> YES_COV
- NO_PERF -> YES_PERF
- NO_PERFBD -> YES_PERFBD
Previously, COV defaulted to yes for tests, and PERFBD defaulted to yes
for benches. This is sometimes useful, but much less often than I
originally thought. Might as well not pay for what we don't use.
With this, the build features of the test/bench runners are consistent
by default, which is probably a good thing.
This _does_ have a noticable, if minor, impact on test runtime:
YES_COV: 674.15s
NO_COV: 584.97s (-13.2%)
As for the naming, the YES_* prefix is needed to avoid conflicts with
the tool variables themselves. I'm not sure what the best approach to
variable naming is here...
$ YES_PERF=1 PERF=~/my_perf/my_perf make test-runner -j
So instead of preprocessing lfs.t.a.c -> lfs.t.c, we preprocess
lfs.t.c -> lfs.t.a.c.
Really the extension should indicate what tool it was generated by, not
what tool it should be consumed by.
The previous commit that changed this states:
> Changed prettyasserts.py rule to .a.c => .c, allowing other .a.c files
> in the future.
But I'm not really sure why we would ever not just run prettyasserts.py
on every C file...
External use of prettyasserts.py made it clear the previous naming was a
bit weird.
Unfortunately, prettyasserts.py is having a hard time keeping up with
the constantly increasing number of tests. This is creating real
friction when debugging, as it now takes ~6x the time to preprocess
asserts as it does to actually compile the thing:
$ time ./scripts/prettyasserts.py \
-a LFS_ASSERT -u LFS_UNREACHABLE \
lfs.t.a.c -o lfs.t.c
real 0m16.187s
user 0m16.163s
sys 0m0.025s
$ time gcc -c -O0 -I. lfs.t.c -o lfs.o
real 0m2.466s
user 0m2.345s
sys 0m0.105s
Externally, I've rewritten prettyasserts.py in Rust, with more attention
towards performance (prettyasserts.py does quite a number of string
allocations). The result is quite satisfying:
$ time ~/prettyasserts/prettyasserts \
-a LFS_ASSERT -u LFS_UNREACHABLE \
lfs.t.a.c -o lfs.t.c
real 0m0.504s
user 0m0.464s
sys 0m0.040s
However, adding Rust as a requirement to test littlefs would be, uh,
quite a big jump.
So instead, littlefs keeps prettyassert.py, so only Python is needed out
of the box, and if the slow preprocessing is too much users are welcome
to provide their own prettyasserts binary via the PRETTYASSERTS env
variable:
$ time \
DEBUG=1 \
make test-runner -j
real 0m22.204s
user 0m44.841s
sys 0m1.478s
$ time \
DEBUG=1 PRETTYASSERTS=~/prettyasserts/prettyasserts \
make test-runner -j
real 0m5.699s
user 0m23.590s
sys 0m1.151s
The main benefit is control over error reporting and avoiding the dive
into stdlib layers when debugging thanks to __builtin_trap().
This changes -p/--pattern -> -a/--assert
And adds -u/--unreachable
I think ctags's defaults may have changed recently or something, but
it's struggling to find function definitions when there's a
predeclaration in the same file.
Adding --fields=+n (line numbers) fixes this.
So now instead of needing:
./scripts/test.py ./runners/test_runner test_dtree
You can just do:
./scripts/test.py test_dtree
Or with an explicit path:
./scripts/test.py -R./runners/test_runner test_dtree
This makes it easier to run the script manually. And, while there may be
some hiccups with the implicit relative path, I think in general this will
make the test/bench scripts easier to use.
There was already an implicit runner path, though only if the test suite
was completely omitted. I'm not sure that would ever have actually
been useful...
---
Also increased the permutation field size in --list-*, since I noticed it
was overflowing.
No more typing this wrong. It probably should have been named make ctags
originally, but since it's been make tags for so long I'm leaving both
as an option for now.
Especially with partial builds of tests (TESTS=tests/t1_rbyd.toml)
becoming more useful, these warning have little value and hide other,
actually-useful warnings.
- Added support for negative numbers in the leb16 encoding with an
optional 'w' prefix.
- Changed prettyasserts.py rule to .a.c => .c, allowing other .a.c files
in the future.
- Updated .gitignore with missing generated files (tags, .csv).
- Removed suite-namespacing of test symbols, these are no longer needed.
- Changed test define overrides to have higher priority than explicit
defines encoded in test ids. So:
./runners/bench_runner bench_dir_open:0f1g12gg2b8c8dgg4e0 -DREAD_SIZE=16
Behaves as expected.
Otherwise it's not easy to experiment with known failing test cases.
- Fixed issue where the -b flag ignored explicit test/bench ids.
- Renamed struct_.py -> structs.py again.
- Removed lfs.csv, instead prefering script specific csv files.
- Added *-diff make rules for quick comparison against a previous
result, results are now implicitly written on each run.
For example, `make code` creates lfs.code.csv and prints the summary, which
can be followed by `make code-diff` to compare changes against the saved
lfs.code.csv without overwriting.
- Added nargs=? support for -s and -S, now uses a per-result _sort
attribute to decide sort if fields are unspecified.
Two flags introduced: -fcallgraph-info=su for stack analysis, and
-ftrack-macro-expansions=0 for cleaner prettyassert.py warnings, are
unfortunately not supported in Clang.
The override vars in the Makefile meant it wasn't actually possible to
remove these flags for Clang testing, so this commit changes those vars
to normal, non-overriding vars. This means `make CFLAGS=-Werror` and
`CFLAGS=-Werror make` behave _very_ differently, but this is just an
unfortunate quirk of make that needs to be worked around.
This is the name expected if you are actually linking against littlefs.
The use as a default build rule is mostly for linting. Most uses of
littlefs likely compile directly with the sources (it is only several K
of code), or use their own build system, and the previous name would have made
linking a bit of a challenge.
Still, this might cause some breakage for someone...
- Changed --(tool)-tool to --(tool)-path in scripts, this seems to be
a more common name for this sort of flag.
- Changed BUILDDIR to not have implicit slash, makes Makefile internals
a bit more readable.
- Fixed some outdated names hidden in less-often used ifdefs.
Based loosely on Linux's perf tool, perfbd.py uses trace output with
backtraces to aggregate and show the block device usage of all functions
in a program, propagating block devices operation cost up the backtrace
for each operation.
This combined with --trace-period and --trace-freq for
sampling/filtering trace events allow the bench-runner to very
efficiently record the general cost of block device operations with very
little overhead.
Adopted this as the default side-effect of make bench, replacing
cycle-based performance measurements which are less important for
littlefs.
This adds -P/--propagate and -Z/--depth to perf.py for showing recursive
results, making it easy to narrow down on where spikes in performance
come from.
This ended up being a bit different from stack.py's recursive results,
as we end up with different (diminishing) numbers as we descend.
This provides 2 things:
1. perf integration with the bench/test runners - This is a bit tricky
with perf as it doesn't have its own way to combine perf measurements
across multiple processes. perf.py works around this by writing
everything to a zip file, using flock to synchronize. As a plus, free
compression!
2. Parsing and presentation of perf results in a format consistent with
the other CSV-based tools. This actually ran into a surprising number of
issues:
- We need to process raw events to get the information we want, this
ends up being a lot of data (~16MiB at 100Hz uncompressed), so we
paralellize the parsing of each decompressed perf file.
- perf reports raw addresses post-ASLR. It does provide sym+off which
is very useful, but to find the source of static functions we need to
reverse the ASLR by finding the delta the produces the best
symbol<->addr matches.
- This isn't related to perf, but decoding dwarf line-numbers is
really complicated. You basically need to write a tiny VM.
This also turns on perf measurement by default for the bench-runner, but at a
low frequency (100 Hz). This can be decreased or removed in the future
if it causes any slowdown.
The main change is requiring field names for -b/-f/-s/-S, this
is a bit more powerful, and supports hidden extra fields, but
can require a bit more typing in some cases.
- Changed multi-field flags to action=append instead of comma-separated.
- Dropped short-names for geometries/powerlosses
- Renamed -Pexponential -> -Plog
- Allowed omitting the 0 for -W0/-H0/-n0 and made -j0 consistent
- Better handling of --xlim/--ylim
Now both scripts also fallback to guessing what fields to use based on
what fields can be converted to integers. This is more falible, and
doesn't work for tests/benchmarks, but in those cases explicit fields
can be used (which is what would be needed without guessing anyways).
These are really just different flavors of test.py and test_runner.c
without support for power-loss testing, but with support for measuring
the cumulative number of bytes read, programmed, and erased.
Note that the existing define parameterization should work perfectly
fine for running benchmarks across various dimensions:
./scripts/bench.py \
runners/bench_runner \
bench_file_read \
-gnor \
-DSIZE='range(0,131072,1024)'
Also added a couple basic benchmarks as a starting point.
- Added the littlefs license note to the scripts.
- Adopted parse_intermixed_args everywhere for more consistent arg
handling.
- Removed argparse's implicit help text formatting as it does not
work with perse_intermixed_args and breaks sometimes.
- Used string concatenation for argparse everywhere, uses backslashed
line continuations only works with argparse because it strips
redundant whitespace.
- Consistent argparse formatting.
- Consistent openio mode handling.
- Consistent color argument handling.
- Adopted functools.lru_cache in tracebd.py.
- Moved unicode printing behind --subscripts in traceby.py, making all
scripts ascii by default.
- Renamed pretty_asserts.py -> prettyasserts.py.
- Renamed struct.py -> struct_.py, the original name conflicts with
Python's built in struct module in horrible ways.
With more scripts generating CSV files this moves most CSV manipulation
into summary.py, which can now handle more or less any arbitrary CSV
file with arbitrary names and fields.
This also includes a bunch of additional, probably unnecessary, tweaks:
- summary.py/coverage.py use a custom fractional type for encoding
fractions, this will also be used for test counts.
- Added a smaller diff output for size scripts with the --percent flag.
- Added line and hit info to coverage.py's CSV files.
- Added --tree flag to stack.py to show only the call tree without
other noise.
- Renamed structs.py to struct.py.
- Changed a few flags around for consistency between size/summary scripts.
- Added `make sizes` alias.
- Added `make lfs.code.csv` rules
These are just some minor quality of life improvements
- Added a "make build-test" alias
- Made test runner a positional arg for test.py since it is almost
always required. This shortens the command line invocation most of the
time.
- Added --context to test.py
- Renamed --output in test.py to --stdout, note this still merges
stderr. Maybe at some point these should be split, but it's not really
worth it for now.
- Reworked the test_id parsing code a bit.
- Changed the test runner --step to take a range such as -s0,12,2
- Changed tracebd.py --block and --off to take ranges
Doing this now specifically because clang does not have
-Wjump-misses-init, but I've been looking for an excuse to remove these
for a while.
These warning flags create more annoyance than they add value. There is
probably a reason they aren't included in -Wall + -Wextra.
-Wshadow specifically is potentially harmful as it forces coming up with
new, sometimes less descriptive names for repeated variables.
Dependent projects should use different flags for their dependencies if
this introduces problems.
The main change here from the previous test framework design is:
1. Powerloss testing remains in-process, speeding up testing.
2. The state of a test, included all powerlosses, is encoded in the
test id + leb16 encoded powerloss string. This means exhaustive
testing can be run in CI, but then easily reproduced locally with
full debugger support.
For example:
./scripts/test.py test_dirs#reentrant_many_dir#10#1248g1g2 --gdb
Will run the test test_dir, case reentrant_many_dir, permutation #10,
with powerlosses at 1, 2, 4, 8, 16, and 32 cycles. Dropping into gdb
if an assert fails.
The changes to the block-device are a work-in-progress for a
lazily-allocated/copy-on-write block device that I'm hoping will keep
exhaustive testing relatively low-cost.
- Renamed explode_asserts.py -> pretty_asserts.py, this name is
hopefully a bit more descriptive
- Small cleanup of the parser rules
- Added recognization of memcmp/strcmp => 0 statements and generate
the relevant memory inspecting assert messages
I attempted to fix the incorrect column numbers for the generated
asserts, but unfortunately this didn't go anywhere and I don't think
it's actually possible.
There is no column control analogous to the #line directive. I thought
you might be able to intermix #line directives to put arguments at the
right column like so:
assert(a == b);
__PRETTY_ASSERT_INT_EQ(
#line 1
a,
#line 1
b);
But this doesn't work as preprocessor directives are not allowed in
macros arguments in standard C. Unfortunately this is probably not
possible to fix without better support in the language.
Christopher Haster