Commit Graph

2099 Commits

Author SHA1 Message Date
Christopher Haster e18cecc3fb scripts: Tweaked some flags in dbgmap.py/tracebd.py
Mostly to move away from unnecessary shortform flags. Using shortform
flags for what is roughly an unbounded enum just causes too many flag
conflicts as scripts grow:

- -r/--read -> --reads
- -p/--prog -> --progs
- -e/--erase -> --erases
- -w/--wear -> --wear

- -i/--in-use -> -%/--usage
- -M/--mdirs -> --mdirs
- -B/--btrees -> --btress
- -D/--datas -> --data/--datas

I may have had too much fun forcing argparse to make -%/--usage to work.
The percent sign caused a lot of problems for argparse internally.
2025-04-16 15:21:34 -05:00
Christopher Haster 9781055d8c scripts: Dropped -N shortform for --no-header
--no-header doesn't really deserve a shortform, and this risks conflicts
with -N/--notes in the future, not to mention any other number of flags
that can start with --no-*.
2025-04-16 15:21:32 -05:00
Christopher Haster 0dbd1561ae scripts: Fixed some issues with -k/--keep-open
- Fixed a NameError in watch.py caused by an outdated variable name
  (renamed paths -> keep_open_paths). Yay for dynamic typing.

- Fixed fieldnames is None issue when csv file is empty.
2025-04-16 15:21:27 -05:00
Christopher Haster 3a290c41ab scripts: Reverted -o/-O to include all by-fields by default
For the same reason we output all field fields by default: Because
machines can process more information than humans can.

Worst case, by fields can still be limited via explicit -b/--by flags.
2025-03-12 21:26:11 -05:00
Christopher Haster f3889d8932 scripts: Adopted Canvas class in plot.py
This should have no noticeable impact on plot.py, but shared classes
have proven helpful for maintaining these scripts.

Unfortunately, this did require some tweaking of the Canvas class to get
things working.

Now, instead of storing things in an internal high-resolution grid,
the Canvas class only keeps track of the most recent character, with
bitmasked ints storing sub-char info.

This makes it so sub-char draws overwrite full characters, which is
necessary for plot.py's axis/data overlap to work.
2025-03-12 21:23:16 -05:00
Christopher Haster 4df90dfa0a scripts: Added --squarify-ratio to treemap[d3].py/codemap[d3].py
Might as well. The internal algorithm already supports this.
2025-03-12 21:23:09 -05:00
Christopher Haster f033a55cc5 scripts: Fixed codemap[d3].py ignoring tiling flags
Not sure how this was missed. Fortunately it's just a small tweak to
make the internal partition function accept tiling args, matching
treemap[d3].py.
2025-03-12 21:21:57 -05:00
Christopher Haster 313696ecf9 scripts: Fixed openio issue where some scripts didn't import os
This only failed if "-" was used as an argument (for stdin/stdout), so
the issue was pretty hard to spot.

openio is a heavily copy-pasted function, so it makes sense to just add
the import os to openio directly. Otherwise this mistake will likely
happen again in the future.
2025-03-12 21:18:51 -05:00
Christopher Haster b2646148c1 scripts: Tweaked -./-p/-P flags in ascii scripts
- -*/--add-char/--chars -> -./--add-char/--chars
- -./--points -> -p/--points
- -!/--points-and-lines -> -P/--points-and-lines

Also fixed an issue in plot.py/Attr where non-list default were failing
to concatenate.
2025-03-12 21:18:15 -05:00
Christopher Haster 5b5745bca9 scripts: treemap.py/codemap.py: Tweaked -L to imply -l
And added the optional --no-label to explicitly opt out.

This is a bit more consistent with treemapd3.py/codemapd3.py's handling
of labels, while still keeping the no-label default. It also makes it
easier to temporarily hide labels when editing commands.
2025-03-12 21:14:37 -05:00
Christopher Haster ff803dfc0f scripts: Added vestigial -:/--dots flag to plot.py
This just makes it easier to jump between plot.py and plotmpl.py.
2025-03-12 21:13:40 -05:00
Christopher Haster 59ffbde3ad scripts: treemap.py/codemap.py: Use parts of name for char defaults
So by default, instead of just using "." for tiles, we use interesting
parts of the tile's name:

- For treemap.py, we use the first character of the last by-field (so
  "lfs.c,lfsr_file_write,1234" -> "1").

- For codemap.py, we use the first character of the non-subsystem part
  of the function name (so "lfsr_file_write" -> "w").

This nice thing about this, is the resulting treemap is somewhat
understandable even without colors:

  $ ./scripts/codemap.py lfs.o lfs_util.o lfs.ci lfs_util.ci -W60 -H8
  code 35528 stack 2440 ctx 636
  ffffffoooffaaaaaaaaaaaacccccccccttttccccrrrrpgffmmrraifmmcss
  ffffffwwwttaaaaaaaaaaaacccccccccttttccccrprrpcscmmoommrrcepp
  ffffffwwwttaaaaaaaaalllcccccccccttttccccrpppccscmmsrmmrrrrss
  ccccssrrfclaaaaanneeasscccccccccgpppccccrpppsgsummstmmrrlfgf
  ccccssrrfccaaaaanneeaaaccccccsaagpppcccccrrrfrrcccrrfiiilucs
  ccccssrrtfcfffffaapplcccccccclssgnnllllcrrffrrrccccifssscmcm
  ccccssrrtrdfffffaapppapcccfffllsgnnllllcrrrffrrcccorfsssicnu

Ok, so maybe the word "somewhat" is doing a lot of heavy lifting...
2025-03-12 21:12:12 -05:00
Christopher Haster 3d53f5393d scripts: Added codemap.py
Like codemapd3.py, but with an ascii renderer.

This is basically just codemapd3.py and treemap.py smooshed together.
It's not the cleanest, but it gets the job done. codemap.py is not
the most critical of scripts.

Unfortunately callgraph and stack/ctx info are difficult (impossible?)
to render usefully in ascii, but we can at least do the script calling,
parsing, namespacing, etc, necessary to create the code cost tilemap.
2025-03-12 21:12:12 -05:00
Christopher Haster 34aa054f18 scripts: codemapd3.py cleanup
Just removing outdated/commented-out code, and TODOs that have now been
taken care of.
2025-03-12 21:12:12 -05:00
Christopher Haster 4ea710f62c scripts: Adopted % modifiers in all attr arguments
This turns out to be extremely useful, for the sole purpose of being
able to specify colors/formats/etc in csv fields (-C'%(fields)s' for
example, or -C'#%(field)06x' for a cooler example).

This is a bit tricky for --chars, but doable with a psplit helper
function.

Also fixed a bug in plot.py where we weren't using dataattrs_ correctly.
2025-03-12 21:12:12 -05:00
Christopher Haster 3d355d7783 scripts: Added -t/--tiny to treemap.py
Even though I think this makes less sense for the ascii-rendering
scripts, it's useful to have this flag around when jumping between
treemap.py and treemapd3.py.

And it might actually make sense sometimes now that -t/--tiny does not
override --to-scale.
2025-03-12 21:12:12 -05:00
Christopher Haster 92ac2a757e scripts: Adopted json -> is_json tweak, avoiding name conflict
This was a humorous name conflict that went unnoticed only because we
lazily import json in read_csv.
2025-03-12 21:12:12 -05:00
Christopher Haster c60301719a scripts: Adopted dat tweak in other scripts
This just makes dat behave similarly to Python's getattr, etc:

- dat("bogus")       -> raises ValueError
- dat("bogus", 1234) -> returns 1234

This replaces try_dat, which is easy to forget about when copy-pasting
between scripts.

Though all of this wouldn't be necessary if only we could catch
exceptions in expressions...
2025-03-12 21:12:12 -05:00
Christopher Haster e780fd40f7 scripts: Added codemapd3.py
Inspired heavily by d3 and brendangregg's flamegraphs, codemapd3.py is
intended to be a powerful high-level code exploring tool.

It's a visual tool, so probably best explained visually:

  $ CFLAGS='-DLFS_NO_LOG -DLFS_NO_ASSERT' make -j
  $ ./scripts/codemapd3.py \
          lfs.o lfs_util.o \
          lfs.ci lfs_util.ci \
          -otest.svg -W1500 -H700 --dark
  updated test.svg, code 35528 stack 2440 ctx 636

And open test.svg in a browser of your choice.

(TODO add a make rule for this)

---

Features include:

- Rendering of code cost in a treemap organized by subsystem (based on
  underscore-separated namespaces), making it relatively easy to see
  where the bulk of our code cost comes from.

- Rendering of the deepest stack/ctx cost as a set of tiles, making it
  relatively easy to see where the bulk of our stack cost comes from.

- Interactive (on mouseover) rendering of callgraph info, showing
  dependencies and relevant stack/ctx costs per-function.

  This currently includes 4 modes:

  1. mode-callgraph - This shows the full callgraph, including all
     children's children, which is effectively all dependencies of that
     function, i.e. the total code cost necessary for that _specific_
     function to work.

  2. mode-deepest - This shows the deepest/hot path of calls from that
     function, which is every child that contributes to the function's
     stack cost.

  3. mode-callees - This shows all functions the current function
     immediately calls.

  4. mode-callers - This shows all functions that call the current
     function.

  And yes, cycles are handled correctly: We show the deepest
  non-cyclical path, but display the measured stack usage as infinite.

For more details see ./scripts/codemapd3.py --help.

---

One particularly neat feature I'm happy about is -t/--tiny, which scales
the resulting image such that 1 pixel ~= 1 byte. This should be useful
for comparing littlefs to other filesystems in a way that is visually
interesting.

- d3 - https://d3js.org
- brendangregg's flamegraphs - https://github.com/brendangregg/FlameGraph
2025-03-12 21:08:23 -05:00
Christopher Haster 021e03cdc9 scripts: treemapd3.py: Tweaked nested rendering, precomputed alpha
This replaces the default colors with colors with precomputed alpha.
They should look the same as long as you don't change the background
color.

The reason for this is I need non-transparent colors for a fork of
treemapd3.py that I am working on. I'm attempting to render some arrows
behind the tiles, and with transparency the result is just too noisy.

---

This then broke the nested rendering, which relied on opacity to make
each layer darker, so I've replaced that with an explicit svg
filter-effect.

The opacity hack was non-linear and kinda ugly past depth >~3, so it
should have eventually been replaced anyways.
2025-03-12 20:14:37 -05:00
Christopher Haster fb03e27baf scripts: Added --no-stats to treemap.py/treemapd3.py
The previous behavior of -N/--no-header still rendering a header when
--title is also provided was confusing. I think this is a better API,
at the minor cost of needing to pass one more flag if you don't want
stats in the header.
2025-03-12 20:07:27 -05:00
Christopher Haster 0d134a2830 scripts: Re-added -q/--quiet to result scripts
I forgot that this is still useful for erroring scripts, such as
stack.py when checking for recursion.

Technically this is possible with -o/dev/null, but that's both
unnecessarily complicated and includes the csv encoding cost for no
reason.
2025-03-12 20:02:19 -05:00
Christopher Haster 675a805164 scripts: Added -! as a short-form for --everything
-!/--everything has been useful enough to warrant a short form flag,
and -! is unlikely to conflict with other flags while also getting the
point across that this is a bit of an unusual option.
2025-03-12 20:02:12 -05:00
Christopher Haster b0976379d7 scripts: Added -i/--internal to ctx.py/structs.py, re-limiting structs.py
This adds -i/--internal to ctx.py and structs.py, which has proven
useful for introspection/debugging. Being able to view the ctx/args of
internal functions is nice, even if they don't actually contribute to
the high-level cost.

This also reverts structs.py to limit to .h files by default, to match
ctx.py, once again relying on dwarf file info. This has been a bit
unreliable in the past, but there's not much else that determines if a
struct is part of the "public interface" in C.

But that's what ctx.py is for.

---

Also fixed an issue where structs appearing in multiple files would have
their sizes added together, which ends up with some pretty confusing
results (sizeof(uint32_t) => 8?).
2025-03-12 20:00:56 -05:00
Christopher Haster 1cc38acc91 scripts: Strip compiler suffixes in result scripts
This can be explicitly disabled with -x/--no-strip in the relevant
scripts, but stripping by default seems to be more useful for composing
results in higher-level scripts. It's better for the result names to be
consistent, even if they don't match the .o symbols exactly.

Note some scripts are unaffected:

- cov.py - gcov doesn't seem to have an option for getting the
  unstripped symbols, so we only output the stripped names.

- structs.py - structs.py deals with struct names, which are notably not
  symbols.
2025-03-12 20:00:21 -05:00
Christopher Haster 9e22167a31 scripts: Re-adopted result prefixes
Now that I'm looking into some higher-level scripts, being able to merge
results without first renaming everything is useful.

This gives most scripts an implicit prefix for field fields, but _not_
by fields, allowing easy merging of results from different scripts:

  $ ./scripts/stack.py lfs.ci -o-
  function,stack_frame,stack_limit
  lfs_alloc,288,1328
  lfs_alloc_discard,8,8
  lfs_alloc_findfree,16,32
  ...

At least now these have better support in scripts with the addition of
the --prefix flag (this was tricky for csv.py), which allows explicit
control over field field prefixes:

  $ ./scripts/stack.py lfs.ci -o- --prefix=
  function,frame,limit
  lfs_alloc,288,1328
  lfs_alloc_discard,8,8
  lfs_alloc_findfree,16,32
  ...

  $ ./scripts/stack.py lfs.ci -o- --prefix=wonky_
  function,wonky_frame,wonky_limit
  lfs_alloc,288,1328
  lfs_alloc_discard,8,8
  lfs_alloc_findfree,16,32
  ...
2025-03-12 19:10:17 -05:00
Christopher Haster aae03be54b scripts: Fixed diff result sorting
This was a bit broken when r was None. Which is unusual, but happens
when rendering added/removed diff results.
2025-03-12 19:10:17 -05:00
Christopher Haster 299e2604c6 scripts: Changed -o/-O to an exclusive operation
So:

  $ ./scripts/code.py lfs.o -o- -q

Becomes:

  $ ./scripts/code.py lfs.o -o-

The original intention of -o/-O _not_ being exclusive (aka table is
still rendered unless disabled with -q/--quiet), was to allow results to
be written to csv files and rendered to tables in a single pass.

But this was never useful. Heck, we're not even using this in our
Makefile right now because it would make the rule dependencies more
complicated than it's worth. Even for long-running result scripts
(perf.py, perfbd.py, etc), most of the work is building that csv file,
the cost of rendering a table in a second pass is negligible.

In every case I've used -o/-O, I've also wanted -q/--quiet, and almost
always forget this on the first run. So might as well make the expected
behavior the actual behavior.

---

As a plus, this let us simplify some of the scripts a bit, by replacing
visibility filters with -o/-O dependent by-fields.
2025-03-12 19:10:17 -05:00
Christopher Haster e71aca65d9 scripts: Adopted default visibility in scripts with complex fields
This makes it so scripts with complex fields will still output all
fields to output csv/json files, while only showing a user-friendly
subset unless -f/--field is explicitly provided.

While internal fields are often too much information to show by default,
csv/json files are expected to go to other scripts, not humans. So more
information is more useful up until you actually hit a performance
bottleneck.

And if you _do_ somehow manage to hit a performance bottleneck, you can
always limit the output with explicit -f/--field flags.
2025-03-12 19:10:17 -05:00
Christopher Haster 051bf66f9a scripts: Tried to handle -d/--diff results consistently
With this, we apply the same result modifiers (exprs/defines/hot/etc) to
both the input results and -d/--diff results. So if both start with the
same format, diffing/hotifying/etc should work as expected.

This is really the only way I can seen -d/--diff results working with
result modifiers in a way that makes sense.

The downside of this is that you can't save results with some complex
operation applied, and then diff while applying the same operation,
since most of the newer operations (hotify) are _not_ idempotent.

Fortunately the two alternatives are not unreasonable:

1. Save results _without_ the operation applied, since the operation
   will be applied to both the input and diff results.

   This is a bit asymmetric, but should work.

2. Apply the operation to the input and then pipe to csv.py for diffing.

This used to "just work" when we did _not_ apply operations to output
csv/json, but this was really just equivalent to 1..

I think the moral of the story is you can solve any problem with enough
chained csv.py calls.
2025-03-12 19:10:17 -05:00
Christopher Haster 2f20f53e90 scripts: csv.py: Reverted define filtering to before expr eval
It's just too unintuitive to filter after exprs.

Note this is consistent with how exprs/mods are evaluated. Exprs/mods
can't reference other exprs/mods because csv.py is only single-pass, so
allowing defines to reference exprs/mods is surprising.

And the solution to needing these sort of post-expr/mod references is
the same for defines: You can always chain multiple csv.py calls.

The reason defines were change to evaluate after expr eval was because
this seemed inconsistent with other result scripts, but this is not
actually the case. Other result scripts simply don't have exprs/mods, so
filtering in fold is the same as filtering during collection. Note that
even in fold, filtering is done _before_ the actual fold/sum operation.

---

Also fixed a recursive-define regression when folding. Counter-
intuitively, we _don't_ want to recursively apply define filters. If we
do the results will just end up too confusing to be useful.
2025-03-12 19:10:17 -05:00
Christopher Haster e851c654c5 scripts: Fixed typo hiding zero-sized results in table renderer
This should either have checked diff_result==None, or we should be
mapping diff_result=None => diff_result_=None. To be safe I've done
both.

This was a nasty typo and I only noticed because ctx.py stopped printing
"cycle detected" for our linked-lists (which are expected to be cyclic).
2025-03-12 19:10:17 -05:00
Christopher Haster 5811b11131 scripts: csv.py: Replaced -l/--label with -I/-B/-F for hidden fields
It felt weird that adding hidden fields required changing existing
flags unrelated to the field you actually want to affect, and the
upper/lower flag thing seems to work well for -s/-S sooo...

- Replaced -l/--label with -B/--hidden-by for by fields that can
  be hidden from the table renderer.

- Added -F/--hidden-field as a similar thing for field fields.

- Better integrated -i/--enumerate into by fields, now these actually
  maintain related order. And of course added a matching
  -I/--hidden-enumerate flag.

The only downside is this is eating a lot of flag names.. But one of the
nice thing about limiting this complexity to csv.py is it avoids these
flag names cluttering up the other result scripts.

---

The -F/--hidden-fields flag I'm not so sure about, since field exprs
can't really reference each other (single pass). But it does provide
symmetry with -B/--hidden-by, and reserves the name in case hidden field
fields are more useful in the future.

Unfortunately it _is_ annoyingly inconsistent with other hidden fields
(-S/--sort, -D/--define, etc) in that it does end up in output csvs...

But this script is already feeling way over-engineered as is.
2025-03-12 19:10:17 -05:00
Christopher Haster 1b8733b3de scripts: csv.py: Fixed issue with exprs not always being typechecked
Kind of a complicated corner case, but this shows up if you try to sort
by fields as numbers and not as strings. In theory this is possible by
creating a hidden sort field with a typed expr:

  $ ./scripts/csv.py test.csv -bi -bfunction -Si=i

But we weren't typechecking sort fields that already exist in the by
fields, since these are usually strings.

This fix is to make sure all exprs are in the typechecked fields, even
if they are already in by fields. There's no real cost to this.

---

Note this version does _not_ typecheck i, and sorts by string:

  $ ./scripts/csv.py test.csv -bi -bfunction -Si

This raises the question, should we always sort by string by default?

I don't think so. It's easy to miss the difference, and a typecheck
error is a lot safer than incorrect sorting.

So this will sort by number, with i as a hidden field:

  $ ./scripts/csv.py test.csv -bfunction -Si

If you want to sort by string with a hidden field, this is still
possible with -l/--label:

  $ ./scripts/csv.py test.csv -bi -lfunction -Si
2025-03-12 19:10:17 -05:00
Christopher Haster 7789714560 scripts: Adopted single folding pass, fixing perf[bd].py -r/--hot issue
There's an ordering issue with hotifying and folding when we have
multiple foldable results with children. This was hard to notice since
most of the recursive scripts have unique results, but it _is_ an issue
for perf.py/perfbd.py, which rely on result folding to merge samples.

The fix is to fold _before_ hotifying.

We could fold multiple times to avoid changing the behavior of the
result scripts, but instead I've just moved the folding in the table
renderer up into the relevant main functions. This means 1. we only fold
once, and 2. folding affects outputted csv/json files.

I'm a bit on the fence about this behavior change, but it is a bit more
consistent with how -r/--hot, -z/--depth, etc, affect both table and
csv/json results consistently.

Maybe we should move towards the table render always reflecting the
csv/json results? Most csv/json usage is with -q/--quiet anyways...

---

This does create a new risk in that the table renderer can hide results
if they aren't folded first.

To hopefully avoid this I've added an assert in the table renderer if it
notices results being hidden.
2025-03-12 19:10:17 -05:00
Christopher Haster 861dc3bd6a scripts: csv.py: Added --help-mods to help explain % modifiers
I guess in addition to its other utilities, csv.py is now also turning
into a sort of man database for some of the more complicated APIs in the
scripts:

  ./csv.py --help
  ./csv.py --help-exprs
  ./csv.py --help-mods

It's a bit minimal, but better than nothing.

Also dropped the %c modifier because this never actually worked.
2025-03-12 19:10:17 -05:00
Christopher Haster b2768becaa scripts: Added -l/--labels to csv.py
This gives csv.py access to a hidden feature in our table renderer used
by some of the other scripts: fields that affect by-field grouping, but
aren't actually printed.

For example, this prevents summing same named functions in different
files, but only shows the function name in the table render:

  $ ./scripts/csv.py lfs.code.csv -bfile -bfunction -lfunction
  function                                size
  lfs_alloc                                398
  lfs_alloc_discard                         31
  lfs_alloc_findfree                        77
  ...

This is especially useful when enumerating results. For example, this
prevents any summing without extra table noise:

  $ ./scripts/csv.py lfs.code.csv -i -bfunction -fsize -lfunction
  function                                size
  lfs_alloc                                398
  lfs_alloc_discard                         31
  lfs_alloc_findfree                        77
  ...

I also tweaked -b/--by field defaults a bit to account to
enumerate/label fields a bit better.
2025-03-12 19:10:17 -05:00
Christopher Haster 748815bb46 scripts: Disentangled -r/--hot and -i/--enumerate
This removes most of the special behavior around how -r/--hot and
-i/--enumerate interact. This does mean -r/--hot risks folding results
if -i/--enumerate is not specified, but this is _technically_ a valid
operation.

For most of the recursive result scripts, I've replaced the "i" field
with separate "z" and "i" fields for depth and field number, which I
think is a bit more informative/useful.

I've also added a default-hidden "off" field to structs.py/ctx.py, since
we have that info available. I considered replacing "i" with this, but
decided against it since non-zero offsets for union members would risk
being confusing/mistake prone.
2025-03-12 19:10:17 -05:00
Christopher Haster ac30a20d12 scripts: Reworked to support optional json input/output
Guh

This may have been more work than I expected. The goal was to allowing
passing recursive results (callgraph info, structs, etc) between
scripts, which is simply not possible with csv files.

Unfortunately, this raised a number of questions: What happens if a
script receives recursive results? -d/--diff with recursive results?
How to prevent folding of ordered results (structs, hot, etc) in piped
scripts? etc.

And ended up with a significant rewrite of most of the result scripts'
internals.

Key changes:

- Most result scripts now support -O/--output-json in addition to
  -o/--json, with -O/--output-json including any recursive results in
  the "children" field.

- Most result scripts now support both csv and json as input to relevant
  flags: -u/--use, -d/--diff, -p/--percent. This is accomplished by
  looking for a '[' as the first character to decide if an input file is
  json or csv.

  Technically this breaks if your json has leading whitespace, but why
  would you ever keep whitespace around in json? The human-editability
  of json was already ruined the moment comments were disallowed.

- csv.py requires all fields to be explicitly defined, so added
  -i/--enumerate, -Z/--children, and -N/--notes. At least we can provide
  some reasonable defaults so you shouldn't usually need to type out the
  whole field.

- Notably, the rendering scripts (plot.py, treemapd3.py, etc) and
  test/bench scripts do _not_ support json. csv.py can always convert
  to/from json when needed.

- The table renderer now supports diffing recursive results, which is
  nice for seeing how the hot path changed in stack.py/perf.py/etc.

- Moved the -r/--hot logic up into main, so it also affects the
  outputted results. Note it is impossible for -z/--depth to _not_
  affect the outputted results.

- We now sort in one pass, which is in theory more efficient.

- Renamed -t/--hot -> -r/--hot and -R/--reverse-hot, matching -s/-S.

- Fixed an issue with -S/--reverse-sort where only the short form was
  actually reversed (I misunderstood what argparse passes to Action
  classes).

- csv.py now supports json input/output, which is funny.
2025-03-12 19:09:43 -05:00
Christopher Haster d90a8e87c4 scripts: Removed clearly unused isinf condition in dat parser 2025-03-11 18:50:06 -05:00
Christopher Haster dcbc195b41 scripts: csv.py: Replaced -b/--by exprs with % modifiers
In addition to providing more functionality for creating -b/--by fields,
this lets us remove strings from the expr parser. Strings had no
well-defined operations and could best be described as an "ugly wart".

Maybe we'll reintroduce string exprs in the future, but for now csv.py's
-f/--field fields will be limited to numeric values.

As an extra plus, no more excessive quoting when injecting new -b/--by
fields.

---

This also fixed sorting on non-field fields, which was apparently
broken. Or at least mostly useless since it was defaulting to string
sorting.
2025-03-11 18:48:27 -05:00
Christopher Haster 5f2ea77c42 scripts: plot[mpl].py: Reworked --add-xticklabel/yticklabel
This adopts the Attr rework for the --add-xticklabel and
--add-yticklabel flags.

Sort of.

These require a bit of special behavior to make work, but should at
least be externally consistent with the other Attr flags.

Instead of assigning to by-field groups, --add-xticklabel/yticklabel
assign to the relevant x/y coord:

  $ ./scripts/plotmpl.py \
          --add-xticklabel='0=zero' \
          --add-yticklabel='100=one-hundred'

The real power comes from our % modifiers. As a special case,
--add-xticklabel/yticklabel can reference the special x/y field, which
represents the current x/y coord:

  $ ./scripts/plotmpl.py --y2 --yticks=5 --add-yticklabel='%(y)d KiB'

Combined with format specifiers, this allows for quite a bit:

  $ ./scripts/plotmpl.py --y2 --yticks=5 --add-yticklabel='0x%(y)04x'

---

Note that plot.py only shows the min/max x/yticks, so plot.py only
accepts indexed --add-xticklabel/yticklabels, and will error if the
assigning variant is used.
2025-03-11 18:22:18 -05:00
Christopher Haster 86f3bad2a4 scripts: Adopted Attr rework in plot.py/plotmpl.py
Unifying these complicated attr-assigning flags across all the scripts
is the main benefit of the new internal Attr system.

The only tricky bit is we need to somehow keep track of all input fields
in case % modifiers reference fields, when we could previously discard
non-data fields.

Tricky but doable.

Updated flags:

- -L/--label -> -L/--add-label
- --colors -> -C/--add-color
- --formats -> -F/--add-format
- --chars -> -*/--add-char/--chars
- --line-chars -> -_/--add-line-char/--line-chars

I've also tweaked Attr to accept glob matches when figuring out group
assignments. This is useful for matching slightly different, but
similarly named results in our benchmark scripts.

There's probably a clever way to do this by injecting new by fields with
csv.py, but just adding globbing is simpler and makes attr assignment
even more flexible.
2025-03-11 18:09:18 -05:00
Christopher Haster 8b04e35ea5 scripts: Tweaked how Attr handles indexed attrs
No more special indexed attrs at the top-level, now all attrs are
indexed, even if assigned to a specific group.

This just makes it so group-specific cycles are possible:

  $ ./scripts/treemap.py -Clfs.c=red -Clfs.c=green
2025-03-11 18:07:09 -05:00
Christopher Haster baa1a1b3a8 scripts: treemap[d3].py: Implemented more flexible labeling/coloring system
Now, instead of specifying a specific field or comma-separated set of
order-defined constants, -L/--add-label, -C/--add-color, and
-./--add-char/--chars accept a by-field group assignment similar to
-L/--label in plotmpl.py.

I also reworked our % modifiers to behave a bit more like printf
modifiers with optional field targets.

It gets a bit complicated, but this ends up extremely flexible:

- Assign to a specific group:

    $ ./scripts/treemap.py -Clfs.c,lfsr_format=orange

- Note this is heirarchical, with more specific groups taking priority:

    $ ./scripts/treemap.py -Clfs.c=blue -Clfs.c,lfsr_format=orange

- We can still get the order-assigned behavior by specifying multiple
  options, but note there is no longer a comma ambiguity! This is useful
  if you want to specify a palette and don't care which dataset gets
  which attr:

    $ ./scripts/treemap.py -Cred -Cgreen -Cblue

- Mix and match:

    $ ./scripts/treemap.py -Cred -Cgreen -Cblue -Clfsr_format=orange

- And with the new % modifiers, we can still use labels stored in a
  field:

    $ ./scripts/treemap.py -L'%(label_field)s'

- -./--add-char/--chars in treemap.py is a bit of a special case. Since
  it only accepts single characters, we can still accept multiple
  options with a single flag without having to worry about ambiguities:

    $ ./scripts/treemap.py -.asdf

  Well, unless you want to include a literal '='. This is possible, but
  a bit messy:

    $ ./scripts/treemap.py -.as -.=== -.df

  Yes that is 3 equal signs... One for argparse, one for the assignment,
  one for the '=' literal.

  This one is minor, but nice for terseness.
2025-03-11 17:29:58 -05:00
Christopher Haster 6a6b74d631 scripts: treemap[d3].py: Show redundant datasets as redundant tiles
A painful lesson learned from plot[mpl].py: we should never implicitly
sum results in a late-stage rendering script. It just makes it way to
easy to accidentally render incorrect/misleading data, while being
difficult to notice.

We should always render redundant results as redundant results.

If the redundant results are an error, this hopefully makes the problem
more obvious to the user. And if the user really does want summed
results, they can always use csv.py as an intermediate step:

  $ ./scripts/treemap.py \
          <(./scripts/csv.py lfs.code.csv -bfile -fsize -q -o-)
          -fsize
2025-03-11 15:19:14 -05:00
Christopher Haster 1c92b7e892 scripts: treemap[d3].py: Squared --squarify, added --rectify
This adds --rectify for a parent-aspect-ratio-preserving --squarify
variant, reverting squarify to try to match the aspect ratio of a
square (1:1).

I can see arguments for both of these. On one hand --squarify makes the
squarest squares, which according to Mark Bruls et al's paper on the
topic is easier visually compare. On the other hand --rectify may be
more visually pleasing and fit into parent tiles better.

d3 allows for any ratio, but at the moment I'm not seeing a strong
reason for the extra parameter.
2025-03-11 15:19:04 -05:00
Christopher Haster 2135c6a003 scripts: Added treemapd3.py
Like treemap.py, but outputting an svg file, which is quite a bit more
useful.

Things svg is _not_:

- A simple vector graphics format

Things svg _is_:

- A surprisingly powerful high-level graphics language.

I might have to use svgs as an output format more often. It's
surprisingly easy to generate graphics without worrying about low-level
rendering details.

---

Aside from the extra flags for svg details like font, padding,
background colors, etc, the main difference between treemap.py and
treemapd3.py is the addition of the --nested mode, which renders a
containing tile for each recursive group (each -b/--by field).

There's no way --nested would've worked in treemap.py. The main benefit
is the extra labels per subgroup, which are already hard enough to read
in treemap.py.

Other than that, treemapd3.py is mostly the same as treemap.py, but with
a resolution that's actually readable.
2025-03-11 14:11:07 -05:00
Christopher Haster d6c909e724 scripts: Added treemap.py
Based on the d3 javascript library (https://d3js.org), treemap.py
renders heirarchical data as ascii art:

  $ ./scripts/treemap.py lfs.code.csv \
          -bfunction -fsize --chars=asdf -W60 -H8
  total 65454, avg 369 +-366.8σ, min 3, max 4990
  aaaassssddddddaaaadddddssddfffaaadfffaassaassfasssdfdfsddfad
  aaaassssddddddaaaadddddssddfffaaadfffaassdfaafasssdfdfsddfsf
  aaaassssddddddaaaafffffssddfffsssdaaaddffdfaadfaaasdfafaasfa
  aaaassssddddddaaaafffffaaaddddsssaassddffdfaaffssfssfsfadffa
  aaaassssffffffssssfffffaaaddddsssaassssffddffffssfdffsadfsad
  aaaassssffffffssssaaaaasssffffddfaassssaaassdaaddadffsadadad
  aaaassssffffffssssaaaaasssffffddfddffddssassdfassadffsadaffa
  aaaassssffffffssssaaaaasssffffddfddffddssassdfaddsdadasfsada

(Normally this is also colored, but you know.)

I've been playing around with d3 to try to better visualize code costs
in littlefs, and it's been quite neat. I figured it would be useful to
directly integrate a similar treemap renderer into our result scripts.

That being said, this ascii rendering is probably too difficult to parse
for any non-trivial data. I'm also working on an svg-based renderer, so
treemap.py is really just for in-terminal previews and an exercise to
understand the underlying algorithms, similar to plot.py/plotmpl.py.
2025-03-11 14:10:21 -05:00
Christopher Haster 2b3fdffe4c Renamed tailp -> ptail
This is mainly just to match pcache.

Which is where I would put ptail if these two didn't have annoyingly
different flush semantics.
2025-02-13 16:07:02 -06:00