I think these cases were mostly just overlooked as the API churned
internally, bmosses/bprout were added and removed, etc.
Shaves off some more code:
code stack ctx
before: 35820 2440 640 (+0.0%)
after: 35784 (-0.1%) 2440 (+0.0%) 640 (+0.0%)
I think this was just overlooked when dropping bmoss/bsprouts.
Dropping bmoss here makes it so file size is always the first leb128 in
the data, which is nice.
Saves a bit of code:
code stack ctx
before: 35864 2440 640
after: 35820 (-0.1%) 2440 (+0.0%) 640 (+0.0%)
In hindsight this was way too fragile.
Explicitly checking for both LFS_TYPE_REG and LFS_type_TRAVERSAL (the 2
in-device types that can have attached bshrubs) solves this and
hopefully prevents lfsr_o_isbshrub from falling out-of-date in the
future.
The downside being a little bit more code:
code stack ctx
before: 35832 2440 640
after: 35864 (+0.1%) 2440 (+0.0%) 640 (+0.0%)
Found by test_traversal_mutation_mroot_split_bshrub_l and
test_traversal_mutation_mroot_split_bshrub_r.
This adds LFSR_TAG_ORPHAN, which simplifies quite a bit of the internal
stickynote handling.
Now that we don't have to worry about conflicts with future unknown
types, we can add whatever types we want internally. One useful one
is LFSR_TAG_ORPHAN, which lets us determine stickynote's orphan status
early (in lfsr_mdir_lookupnext and lfsr_mdir_namelookup):
- non-orphan stickynotes -> LFSR_TAG_STICKYNOTE
- orphan stickynotes -> LFSR_TAG_ORPHAN
This simplifies all the places where we need to check if a stickynote
really exists, which is most of the high-level functions.
One downside is that this makes stickynote _manipulation_ a bit more
delicate. lfsr_mdir_lookup(LFSR_TAG_ORPHAN) no longer works as expected,
for example.
Fortunately we can sidestep this issue by dropping down to
lfsr_rbyd_lookup when we need to interact with stickynotes directly,
skipping the is-orphan checks.
---
Saves a nice bit of code:
code stack ctx
before: 35984 2440 640
after: 35832 (-0.4%) 2440 (+0.0%) 640 (+0.0%)
It got a little muddy since this now include the unknown-type changes,
but here's the code diff from before we exposed LFSR_TYPE_STICKYNOTE to
users:
code stack ctx
before: 35740 2440 640
after: 35832 (+0.3%) 2440 (+0.0%) 640 (+0.0%)
Now that name tags are a special case, using a switch case statement
here continues to make less sense.
Also switched to just checking count >= 0 directly instead of via
lfsr_attr_dtag, because lfsr_tag_suptype(lfsr_tag_dtag(rattr)) would've
been a mouthful.
Saves a teensy bit of code:
code stack ctx
before: 35992 2440 640
after: 35984 (-0.0%) 2440 (+0.0%) 640 (+0.0%)
This drops the requirement that all file types are introduced with a
related wcompat flag. Instead, the wcompat flag is only required if
modification _would_ leak resources, and we treat unknown file types as
though they are regular files.
This allows modification of unknown file types without the risk of
breaking anything.
To compare with before the unknown-type rework:
Before:
> Unknown file types are allowed and may leak resources if modified,
> so attempted modification (rename/remove) will error with
> LFS_ERR_NOTSUP.
Now:
> Unknown file types are allowed but must not leak resources if
> modified. If an unknown file type would leak resources, it should set
> a related wcompat flag to only allow mounting RDONLY.
Note this includes directories, which can leak bookmarks if removed, so
filesystems using directories should set the LFSR_WCOMPAT_DIR flag.
But we no longer need the LFSR_WCOMPAT_REG/LFSR_WCOMPAT_STICKYNOTE
flags.
---
The real tricky part was getting lfsr_rename to work with unknown types,
as this broke the invariant that we only ever commit tags we know about.
Fixing this required:
- Fetching the non-unknown-mapped tag in lfsr_rename
- Mapping all name tags to LFSR_TAG_NAME in lfsr_rbyd_appendrattr_
- Adopting LFSR_RATTR_NAME for bookmark name tags
This was broken by the above lfsr_rbyd_appendrattr_ change, but it's
probably good to handle these the same as other name tags anyways.
This adds a bit of code, but not enough that I think this isn't worth
it (or worth a build-time option):
code stack ctx
before: 35924 2440 640
after: 35992 (+0.0%) 2440 (+0.0%) 640 (+0.0%)
This changes how we approach unknown file types.
Before:
> Unknown file types are allowed and may leak resources if modified,
> so attempted modification (rename/remove) will error with
> LFS_ERR_NOTSUP.
Now:
> Unknown file types are only allowed in RDONLY mode. This avoids the
> whole leaking resources headache.
Additionally, unknown types are now mapped to LFS_TYPE_UNKNOWN, instead
of just being forwarded to the user. This allows us to add internal
types/tags to the LFSR_TAG_NAME type space without worrying about
conflicts with future types:
- reg -> LFS_TYPE_REG
- dir -> LFS_TYPE_DIR
- stickynote -> LFS_TYPE_STICKYNOTE
- everything else -> LFS_TYPE_UNKNOWN
Thinking about potential future types, it seems most (symlinks,
compressed files, etc) can be better implemented via custom attributes.
Using custom attributes doesn't mean the filesystem _can't_ inject
special behavior, and custom attributes allow for perfect backwards
compatibility.
So with future types less likely, forwarding type info to users is less
important (and potentially error prone). Instead, allowing on-disk +
internal types to be represented densely is much more useful.
And it avoids setting an upper bound on future types prematurely.
---
This also includes a minor rcompat/wcompat rework. Since we're probably
going to end up with 32-bit rcompat flags anyways, might as well make
them more human-readable (nibble-aligned):
LFS_RCOMPAT_NONSTANDARD 0x00000001 Non-standard filesystem format
LFS_RCOMPAT_WRONLY 0x00000002 Reading is disallowed
LFS_RCOMPAT_BMOSS 0x00000010 Files may use inlined data
LFS_RCOMPAT_BSPROUT 0x00000020 Files may use block pointers
LFS_RCOMPAT_BSHRUB 0x00000040 Files may use inlined btrees
LFS_RCOMPAT_BTREE 0x00000080 Files may use btrees
LFS_RCOMPAT_MMOSS 0x00000100 May use an inlined mdir
LFS_RCOMPAT_MSPROUT 0x00000200 May use an mdir pointer
LFS_RCOMPAT_MSHRUB 0x00000400 May use an inlined mtree
LFS_RCOMPAT_MTREE 0x00000800 May use an mdir btree
LFS_RCOMPAT_GRM 0x00001000 Global-remove in use
LFS_WCOMPAT_NONSTANDARD 0x00000001 Non-standard filesystem format
LFS_WCOMPAT_RDONLY 0x00000002 Writing is disallowed
LFS_WCOMPAT_REG 0x00000010 Regular file types in use
LFS_WCOMPAT_DIR 0x00000020 Directory file types in use
LFS_WCOMPAT_STICKYNOTE 0x00000040 Stickynote file types in use
LFS_WCOMPAT_GCKSUM 0x00001000 Global-checksum in use
---
Code changes:
code stack ctx
before: 35928 2440 640
after: 35924 (-0.0%) 2440 (+0.0%) 640 (+0.0%)
Now that LFS_TYPE_STICKYNOTE is a real type users can interact with, it
makes sense to group it with REG/DIR. This also has the side-effect of
making these contiguous.
---
LFSR_TAG_BOOKMARKs, however, are still hidden from the user. This
unfortunately means there will be a bit of a jump if we ever add
LFS_TYPE_SYMLINK in the future, but I'm starting to wonder if that's the
best way to approach symlinks in littlefs...
If instead LFS_TYPE_SYMLINKS were implied via custom attribute, you
could avoid the headache that comes with adding a new tag encoding, and
allow perfect compatibility with non-symlink drivers. Win win.
This seems like a better approach for _all_ of the theoretical future
types (compressed files, device files, etc), and avoids the risk of
oversaturating the type space.
---
This had a surprising impact on code for just a minor encoding tweak. I
guess the contiguousness pushed the compiler to use tables/ranges for
more things? Or maybe 3 vs 5 is just an easier constant to encode?
code stack ctx
before: 35952 2440 640
after: 35928 (-0.1%) 2440 (+0.0%) 640 (+0.0%)
This adds stickynotes as a target type for most of the test_attrs tests.
These were already parameterized for reg + dir + root, but they did need
some tweaks to allow us to keep files open for most of the test.
I don't really see a use case for this feature, but it keeps the API
consistent. The file name itself is also sort of an attr, and the whole
point of stickynotes is to have something to attach the file name to, so
attaching other attrs makes sense I guess?
Seems like a better name now that LFS_TYPE_STICKYNOTE is its own file
type.
Though this does contain some tests that I think don't even use
stickynotes...
This adds the LFS_TYPE_STICKYNOTE type, allowing users to interact with
stickynotes as long as they aren't orphaned.
This hopefully solves the long-standing mess that was the LFS_O_EXCL
API.
---
As for what I mean by orphaned vs non-orphaned stickynotes:
Non-orphaned stickynotes represent files that have been "created" (via
LFS_O_CREAT), but not "committed" (via sync/close). You can still close
and convert the stickynote to a reg file, so these aren't orphans. These
are also called "uncreated" files in some parts of the codebase:
- open+O_CREAT -> non-orphaned stickynote (uncreated file)
Orphaned stickynotes are possible by either removing an open file, or
desyncing a file before sync/close. These are still invisible to the
user and will be eventually cleaned up after the last file handle is
closed:
- open+remove -> orphaned stickynote (zombied file)
- open+O_CREAT+desync+close -> orphaned stickynote (orphaned file)
Desynced files are a bit special. Even though they technically aren't
orphaned, they also behave like orphaned file handles:
- open+O_CREAT+close -> orphaned stickynote (desynced file)
The idea is this mimics the state of files post-close, and allows for
some tricks like using a desync file as a temporary file with no
observable effects on the filesystem.
---
The motivation for this comes from staring at the LFS_O_EXCL API for too
long and realizing the problem is that littlefs's API contradicts itself
when it comes to whether or not uncreated files exist.
This solution is to consistently treat uncreated files as though they
exist (the alternative would make LFS_O_EXCL pretty much useless), but I
really didn't want to do this as having what appears to be normal files
disappear after powerloss risks confusion.
The compromise here is to give these files a special type, repurposing
the internal LFS_TAG_STICKYNOTE, which hopefully hints to the user these
won't behave like normal files.
If the user is more interested in POSIX compatibility, they can always
map these to either LFS_TYPE_REG or LFS_ERR_NOENT, whichever they think
is the least confusing.
As a quirk of littlefs's API, stickynotes should never actually contain
any data, and will always have size 0.
However they can have custom attributes assigned now (which is I guess
ok? also TODO should probably test this).
---
The implementation right now is a bit naive, I mostly just wanted to get
the tests working again in this new model. It may be possible to claw
back some of this code cost:
code stack ctx
before: 35740 2440 640
after: 35952 (+0.6%) 2440 (+0.0%) 640 (+0.0%)
This may be useful for compression in the future, where compression +
noise can result in blocks _larger_ than the expected weight.
Thinking about how compression might be integrated into littlefs, it
would be nice if such a topology did _not_ trigger asserts. This would
allow littlefs images to interact with compressed files at least a
little bit (rename/remove could be very useful), even if the compression
algorithm isn't supported.
Supporting this requires only a single clamp in lfsr_file_lookupleaf,
but it's a little bit more costly than you might expect:
code stack ctx
before: 35692 2440 640
after: 35740 (+0.1%) 2440 (+0.0%) 640 (+0.0%)
This is due to internal API awkwardness:
1. LFSR_DATA_TRUNCATE is surprisingly costly
2. We need to create a local weight copy in case the caller's is NULL
- test_fwrite_reversed_litmus_fragments
- test_fwrite_reversed_litmus_blocks
- test_fwrite_freversed
- test_fwrite_freversed_litmus_fragments
- test_fwrite_freversed_litmus_blocks
- test_fwrite_truncate_pos
- test_fwrite_fruncate_pos
And hey, they found some bugs:
- crystal_thresh=-1 was broken due to integer overflow in some signed
math.
Fortunately when crystal_thresh=-1 we can just skip the crystal
lookups entirely. This saves a btree lookup in fully-fragmented files.
- We were including empty fragments in our crystal size, when we should
only use them to determine crystal boundaries, like bptrs.
This is a common case for the first entry in a sparse file.
- We weren't updating pos on fruncate. fruncate's effect on pos was
actually not tested at all.
Which raises the question, what should the behavior be? Match
lfsr_file_truncate and leave the pos unaffected?
I ended up having fruncate update the file pos to keep the same pos
relative to the end, as I figured this would have the least surprise
for users. So lfsr_file_read should return the same bytes unless
clobbered.
This is almost a mirror image of lfsr_file_truncate, except we don't
allow negative positions, so fruncating more than pos forces pos to 0.
---
This behavior is now covered in a couple tests:
- test_fwrite_truncate_pos
- test_fwrite_fruncate_pos
- test_fwrite_freversed
- test_fwrite_freversed_litmus_fragments
- test_fwrite_freversed_litmus_blocks
Code changes:
code stack ctx
before: 35688 2440 640
after: 35692 (+0.0%) 2440 (+0.0%) 640 (+0.0%)
littlefs is not a C++ project, and it's important to make sure users are
aware of that in case the header file ever breaks C++ (C++ is _not_
compatible with C99).
So dropping these guards.
C++ users should wrap the relevant includes with extern "C":
extern "C" {
#include "lfs.h"
}
So instead of:
CFLAGS='-DLFS_YES_REVDBG=1' make
You can just do:
LFS_YES_REVDBG=1 make
I've been hesitant to add this, as I've never seen this pattern in
another project (why?), but it's just too convenient to not give it a
try.
This lets you specify mount/format flags globally, via -DLFS_YES_REVDBG,
for example.
In addition to the convenience of not needing to edit code, these flags
may also be able to reduce code cost by eliminating the various flag
checks and untaken code paths.
At the moment this relies on dead code elimination via the lfsr_m_is*
functions, to keep the codebase from exploding too much.
This tweaks a number of extended revision count things:
- Added LFS_REVDBG, which adds debug info to revision counts.
This initializes the bottom 12 bits of every revision count with a
hint based on rbyd type, which may be useful when debugging:
- 68 69 21 v0 (hi!.) => mroot anchor
- 6d 72 7e v0 (mr~.) => mroot
- 6d 64 7e v0 (md~.) => mdir
- 62 74 7e v0 (bt~.) => file btree node
- 62 6d 7e v0 (bm~.) => mtree node
This may be overwritten by the recycle counter if it overlaps, worst
case the recycle counter takes up the entire revision count, but these
have been chosen to at least keep some info if partially overwritten.
To make this work required the LFS_i_INMTREE hack (yay global state),
but a hack for debug info isn't the end of the world.
Note we don't have control over data blocks, so there's always a
chance they end up containing what looks like one of the above
revision counts.
- Renamed LFS_NOISY -> LFS_REVNOISE
- LFS_REVDBG and LFS_REVNOISE are incompatible, so using both asserts.
This also frees up the theoretical 0x00000030 state for an additional
rev mode in the future.
- Adopted LFS_REVNOISE (and LFS_REVDBG) in btree nodes as well.
If you need rev noise, you probably want it in all rbyds/metadata
blocks, not just mdirs.
---
This had no effect on the default code size, but did affect
LFS_REVNOISE:
code stack ctx
before: 35688 2440 640
after: 35688 (+0.0%) 2440 (+0.0%) 640 (+0.0%)
revnoise before: 35744 2440 640
revnoise after: 35880 (+0.4%) 2440 (+0.0%) 640 (+0.0%)
default: 35688 2440 640
revdbg: 35912 (+0.6%) 2448 (+0.3%) 640 (+0.0%)
revnoise: 35880 (+0.5%) 2440 (+0.0%) 640 (+0.0%)
This is based off the parity impl in Sean Eron Anderson's Bit Twiddling
Hacks, who attributes the idea to Mathew Hendry.
Basically the idea is to encode a small lookup table in an integer, and
extract using a shift + mask:
.-- LFSR_TAG_MASK0
.|-- LFSR_TAG_MASK2
.||-- LFSR_TAG_MASK8
.|||-- LFSR_TAG_MASK12
vvvv
0x0fff & (-1U << ((0xc820 >> (4*((tag >> 12) & 0x3))) & 0xf))
'--.-' ^ '--------.--------'
key mask gcc complains w/o this mask bits
Saves a bit of code at the cost of some stack. I guess because GCC is
trying to avoid multiple constant pool lookups? This may just be
compiler noise:
code stack ctx
before: 35692 2432 640
after: 35688 (-0.0%) 2440 (+0.3%) 640 (+0.0%)
- Gave lfs_parity its own backup implementation.
Since these are static inline functions, shared implementations don't
matter as much here, so why do more work than we have to.
Save a bit of code too:
code stack ctx
yes-builtins: 35692 2432 640
no-builtins before: 35996 (-0.9%) 2504 (+3.0%) 640 (+0.0%)
no-builtins after: 35960 (-0.8%) 2504 (+3.0%) 640 (+0.0%)
Though maybe this is an argument for these functions not being static
inline...
- Tweaked lfs_popc for readability (the 7-digit mask was annoying me).
- Added a link to Sean Eron Anderson's Bit Twiddling Hacks page:
https://graphics.stanford.edu/~seander/bithacks.html
These have been published as public domain, so I don't think this is
strictly necessary, but the page is a great resource and deserves
mention.
Instead, make codemap/codemap-tiny just generate the relevant .svgs:
- dropped make codemap
- dropped make stackmap
- dropped make ctxmap
- make codemap-svg -> make codemap
- make codemap-tiny-svg -> make codemap-tiny
The ascii-art codemaps just really aren't useful due to their low
resolution. We might as well repurpose the relevant make rules to save
keystrokes.
Though I did keep the ascii-art as a step in make codemap/codemap-tiny,
just for fun.
Velociraptors inbound.
This eliminates dags (directed acyclic graphs) from file bshrubs/btrees,
which were the only source of dags in the filesystem. This means
littlefs is now strictly a pure tree, in that no blocks have more than
one parent (ignoring in-RAM references!).
Up until this point, dags could be created in file bshrubs/btrees via
random writes that place fragments in the middle of a block:
.-------------. .-------------------.
| aaaaaaaaaaa | -> | aaaaa | b | aaaaa |
'-------------' '-------------------'
| | v |
v | .-. |
.-------------. | |b| |
| aaaaaaaaaaa | v '-' v
'-------------' .-------------.
| aaaaaaaaaaa |
'-------------'
Now, fragments that would create dags instead trigger block
recrystallization, rewriting the left sibling into a new block if
necessary:
.-------------. .----------------.
| aaaaaaaaaaa | -> | aaaaab | aaaaa |
'-------------' '----------------'
| | '-.
v v v
.-------------. .--------. .-------.
| aaaaaaaaaaa | | aaaaab | | aaaaa |
'-------------' '--------' '-------'
Allowing dags was great for random-write performance, but it creates
problems for future planned features:
1. Current plans for more advanced block allocators rely on blocks only
having one parent. Otherwise it's difficult to know which reference
is the last reference to a block.
2. Dags create a really funny problem for error correction via block
redundancy. Naively, if you try to repair blocks every time you
encounter a given block error, you will end up exploding the block
into n copies, 1 for every parent. Not great!
---
Eliminating these dags was a bit... tricky...
Originally I was planning to just alloc/rewrite blocks in
lfsr_file_carve, but it turns out we can make lfsr_file_flush_ do all
the work with an extra would-dag checks. Handling dags in
lfsr_file_flush_ also gives us a chance to merge any pending data and
get the most out of the block rewrite.
This does give us a bit of technical debt in that we will probably still
need the block splitting in lfsr_file_carve for future features
(advanced hole APIs, alternative write strategies, etc), but it's
probably worth it for code savings in the default build.
Unfortunately this does add to the mess that is lfsr_file_flush_'s
control flow graph:
lfsr_file_flush_
|
v
.--> lookup left crystal .--> lookup left sibling <-.
| | | | |
| v | v |
| erased? | dag? (new!) |
| .---------y n | .---------y n |
| | v | | v |
| | lookup right crystal | | lookup right sibling |
| | | | | | |
| | v | | v |
| | >=crystal_thresh? | | coalesce |
| | y n------------' | | |
| | v | v |
| | lookup left neighbor | carve-----------'
| | | |
| | v |
| | erased? |
| +---------y n |
| | v |
| | alloc <---+-------'
| | | |
| | v |
| '---> crystallize |
| | |
| v |
| good? |
| y n------'
| v
'----------carve
I did scratch my head for a bit trying to think if there was a better
way to organize this, but came up empty.
It looks complicated, but we really only have two* loops (ignoring the
relocation loop): One that crystallizes blocks, and one that coalesces
fragments. The problem is that we end jumping between the two depending
on what we find in the btree.
In a sane system, this would be implemented as mutually recursive
functions, but this is littlefs, the whole point is that we don't use
recursion.
---
The good news is that this added surprisingly little code (and saved
stack?):
code stack ctx
before: 35600 2448 640
after: 35692 (+0.3%) 2432 (-0.7%) 640 (+0.0%)
- Trying to prefer crystal over compact verbiage to try to avoid
confusion with metadata/rbyd compaction
- crystal_thresh >= block_size implying a fully-fragmented file was a
mistake, it should be crystal_thresh > block_size.
crystal_thresh == block_size has the behavior of waiting until the
last moment to crystallize a block, but this still breaks the
fully-fragmented random-write guarantee.
This changed during development, so the comment was probably just
outdated.
This was caused by including the shrub bit in the tag comparison in
Rbyd.lookup.
Fixed by adding an extra key mask (0xfff). Note this is already how
lfsr_rbyd_lookup works in lfs.c.
Bit of a silly, but problematic, bug, probably introduced during the
various lfsr_bptr_t/lfsr_data_t reworks, but basically we never actually
fragmented the last fragment in a bptr.
We were fragmenting all fragments in a bptr _above_ fragment_size, but
then we'd stop at the last fragment and keep it around as a bptr,
completely wasting all of the work to fragment the block. The reason for
the different behavior being that we can combine the last fragment with
the carved data to avoid an additional commit.
Fortunately the solution is pretty non-invasive. We can just assume any
bptrs <= fragment_size should be written out as fragments.
Added test_fwrite_truncate_litmus_fragment and
test_fwrite_fruncate_litmus_fragment to catch this in the future.
Code changes:
code stack ctx
before: 35588 2448 640
after: 35600 (+0.0%) 2448 (+0.0%) 640 (+0.0%)
- Fixed Mtree.lookupleaf accepting mbid=0, which caused dbglfs.py to
double print all files with mbid=-1
- Fixed grm mids not being mapped to mbid=-1 and related orphan false
positives
I've made this mistake before!
One would think that it would be more interesting to show progs over
erases when they overlap, since progs always subset erases and show more
detail. However, erases occur much more rarely and are usually followed
by progs, so when rendering is low resolution (ascii) it's easy for
progs to completely cover up all erase operations.
Prioritizing erases prevents this.
At least this nuance is better documented this time around.
- dropped lfsr_btree_commitleaf
- dropped lfsr_bshrub_commitleaf
- dropped lfsr_file_commitleaf
The problem is that, thanks to rbyd compactions/splits/merges/etc, we
end up leaving the leaf rbyd in a more-or-less undefined state.
I was trying to adopt commitleaf in lfsr_file_carve, the function with
the most glaring potential for commitleaf, but the leaf rbyd behavior is
extremely error prone and requires quite a bit of extra circuitry to use
correctly.
The end result looked like it would need more code, more stack
(lfsr_file_carve _is_ on the stack hot path), for a minor speed
improvement. So I decided to drop the idea. We can probably expect file
carving to be dominated by progs/erases anyways.
lfsr_btree_commit_ still needs to lookup parent rbyds, so it would have
only saved ~1 out of O(log_b n) btree lookups (though this may still be
significant given the ridiculous branching factor of btrees).
---
But it _is_ interesting to note that there is still potential
performance savings on the floor if we didn't care about code size.
Without necessarily sacrificing our bounded RAM constraint.
I could imagine a build in the future that prioritizes performance over
code size by strictly using leaf rbyd functions, iterating over leaf
rbyds before iterating the parent btree, etc.
But that's the future. Simply getting things working is the priority
right now.
---
Saves a bit of code/stack:
code stack ctx
before: 35600 2456 640
after: 35588 (-0.0%) 2448 (-0.3%) 640 (+0.0%)
Note that lookupleaf is still useful for the case where bids have
multiple attrs attached (none so far, but the plan is for the dedup tree
to leverage this).
This should have been updated when we dropped becksums (way back in
5fa85583!), we only ever need at most 3 rattrs to complete a carve
operation (left sibling, rattr, right sibling).
Just a free 24 byte stack savings sitting right there:
code stack ctx
before: 35600 2480 640
after: 35600 (+0.0%) 2456 (-1.0%) 640 (+0.0%)
So now crystal_thresh only controls when fragments are compacted into
blocks, while fragment_thresh controls when blocks are broken into
fragments. Setting fragment_thresh=-1 will follow crystal_thresh and
keeps the previous behavior.
These were already two separate pieces of logic, so it makes sense to
provide two separate knobs for tuning.
Setting fragment_thresh lower than crystal_thresh has some potential to
reduce hysteresis in cases where random writes push blocks close to
crystal_thresh. It will be interesting to explore this more when
benchmarking.
---
The additional config option adds a bit of code/ctx, but hopefully that
will go away in the future config rework:
code stack ctx
before: 35584 2480 636
after: 35600 (+0.0%) 2480 (+0.0%) 640 (+0.6%)
This lets us cram in one more mask for potential redund bits:
name tag mask
LFSR_TAG_MASK0 0x0000 0x0fff ---- 1111 1111 1111
LFSR_TAG_MASK2 0x1000 0x0ffc ---- 1111 1111 11--
LFSR_TAG_MASK8 0x2000 0x0f00 ---- 1111 ---- ----
LFSR_TAG_MASK12 0x3000 0x0000 ---- ---- ---- ----
'.-' '.-' '---.---'
mode bits -' | | ^
suptype ------' | |
subtype --------------' |
redund bits ------------------'
I toyed around with a bitwise alternative to the lookup table, but
couldn't come up with anything simpler than these:
- 0xfff & ~((((1<<((i>>1)*8))-1) << ((i&1)*4)) | ((1<<(i*2))-1))
- 0xfff & ~((1 << (((i>>1)*8)+((i&1)<<(1+(i>>1)))))-1)
- 0xfff & ~((1<<(2*i*i))-1) (requires multiply and 32-bit shift)
---
This also replaces the mdir/rbyd/btree/mtree lookup/sublookup/suplookup
functions with a single flexible lookup function that accepts tag masks.
This ended up adding a bit of code/stack (the extra NULL args are
surprisingly pricey), but will hopefully make the redund bits
easier/cheaper to use:
code stack ctx
before: 35548 2472 636
after: 35584 (+0.1%) 2480 (+0.3%) 636 (+0.0%)
This was a surprising side-effect the script rework: Realizing the
internal btree/rbyd lookup APIs were awkwardly inconsistent and could be
improved with a couple tweaks:
- Adopted lookupleaf name for functions that return leaf rbyds/mdirs.
There's an argument this should be called lookupnextleaf, since it
returns the next bid, unlike lookup, but I'm going to ignore that
argument because:
1. A non-next lookupleaf doesn't really make sense for trees where
you don't have to fetch the leaf (the mtree)
2. It would be a bit too verbose
- Adopted commitleaf name for functions that accept leaf rbyds.
This makes the lfsr_bshrub_commit -> lfsr_btree_commit__ mess a bit
more readable.
- Strictly limited lookup and lookupnext to return rattrs, even in
complex trees like the mtree.
Most use cases will probably stick to the lookupleaf variants, but at
least the behavior will be consistent.
- Strictly limited lookup to expect a known bid/rid.
This only really matters for lfsr_btree/bshrub_lookup, which as a
quirk of their implementation _can_ lookup both bid + rattr at the
same time. But I don't think we'll need this functionality, and
limited the behavior may allow for future optimizations.
Note there is no lfsr_file_lookup. File btrees currently only ever
have a single leaf rattr, so this API doesn't really make sense.
Internal API changes:
- lfsr_btree_lookupnext_ -> lfsr_btree_lookupleaf
- lfsr_btree_lookupnext -> lfsr_btree_lookupnext
- lfsr_btree_lookup -> lfsr_btree_lookup
- added lfsr_btree_namelookupleaf
- lfsr_btree_namelookup -> lfsr_btree_namelookup
- lfsr_btree_commit__ -> lfsr_btree_commit_
- lfsr_btree_commit_ -> lfsr_btree_commitleaf
- lfsr_btree_commit -> lfsr_btree_commit
- added lfsr_bshrub_lookupleaf
- lfsr_bshrub_lookupnext -> lfsr_bshrub_lookupnext
- lfsr_bshrub_lookup -> lfsr_bshrub_lookup
- lfsr_bshrub_commit_ -> lfsr_bshrub_commitleaf
- lfsr_bshrub_commit -> lfsr_bshrub_commit
- lfsr_mtree_lookup -> lfsr_mtree_lookupleaf
- added lfsr_mtree_lookupnext
- added lfsr_mtree_lookup
- added lfsr_mtree_namelookupleaf
- lfsr_mtree_namelookup -> lfsr_mtree_namelookup
- added lfsr_file_lookupleaf
- lfsr_file_lookupnext -> lfsr_file_lookupnext
- added lfsr_file_commitleaf
- lfsr_file_commit -> lfsr_file_commit
Also added lookupnext to Mdir/Mtree in the dbg scripts.
Unfortunately this did add both code and stack, but only because of the
optional mdir returns in the mtree lookups:
code stack ctx
before: 35520 2440 636
after: 35548 (+0.1%) 2472 (+1.3%) 636 (+0.0%)
The exception being LFS_DEBUG. A bit inconsistent, but the at least
consistent with LFS_ERR* vs LFS_ERROR, and may help reduce name
conflicts:
- LFS_DEBUGRBYDFETCHES -> LFS_DBGRBYDFETCHES
- LFS_DEBUGRBYDBALANCE -> LFS_DBGRBYDBALANCE
- LFS_DEBUGRBYDCOMMITS -> LFS_DBGRBYDCOMMITS
- LFS_DEBUGBTREEFETCHES -> LFS_DBGBTREEFETCHES
- LFS_DEBUGBTREECOMMITS -> LFS_DBGBTREECOMMITS
- LFS_DEBUGMDIRFETCHES -> LFS_DBGMDIRFETCHES
- LFS_DEBUGMDIRCOMMITS -> LFS_DBGMDIRCOMMITS
- LFS_DEBUGALLOCS -> LFS_DBGALLOCS
So mbid=0 now implies the mdir is not inlined.
Downsides:
- A bit more work to calculate
- May lose information due to masking everything when mtree.weight==0
- Risk of confusion when in-lfs.c state doesn't match (mbid=-1 is
implied by mtree.weight==0)
Upsides:
- Includes more information about the topology of the mtree
- Avoids multiple dbgmbids for the same physical mdir
Also added lfsr_dbgmbid and lfsr_dbgmrid to help make logging
easier/more consistent.
And updated dbg scripts.
- mdir_bits -> mbits
- lfsr_mid_bid -> lfsr_mbid
- lfsr_mid_rid -> lfsr_mrid
These now match the naming in the dbg scripts.
I feel like this is more terse in a way that is also more readable, but
maybe that's just me.
So now:
(block_size)
mbits = nlog2(----------) = nlog2(block_size) - 3
( 8 )
Instead of:
( (block_size))
mbits = nlog2(floor(----------)) = nlog2(block_size & ~0x7) - 3
( ( 8 ))
This makes the post-log - 3 formula simpler, which we probably want to
prefer as it avoids a division. And ceiling is arguably more intuitive
corner case behavior.
This may seem like a minor detail, but because mbits is purely
block_size derived and not configurable, any quirks here will become
a permanent compatibility requirement.
And hey, it saves a couple bytes (I'm not really sure why, the division
should've been optimized to a shift):
code stack ctx
before: 35528 2440 636
after: 35520 (-0.0%) 2440 (+0.0%) 636 (+0.0%)
Mainly adopting the added flexibility in csv.py, also adding make
codemap-svg and friends for code map generation:
- Split result commands into separate result, result-csv, and
result-diff commands so csv generation is explicit.
So make result no longer implicitly overwrites csv files:
make code
make code-csv -.
make code |
make code-diff <'
This gives more control over result diffing.
make code-csv _is_ more or less just a dependency on the lfs.code.csv
rule, but it avoids BUILDDIR mess and is easier to remember.
- Added make codemap/stackmap/ctxmap for in-terminal code/stack/ctx
ascii art.
I was a bit on the fence on these, since the result is more pretty
than useful, but eh, can always drop them in the future.
- Added make codemap-svg/codemap-tiny-svg for generating interactive
codemap svgs.
This raised an interesting question if the make commands should
generate light or dark mode svgs. I settled on dark mode since that's
what I personally find the most useful.
I think the way this will breakdown is with dark mode generally used
for development, and light mode generally used for published material.
And it's not too hard to run the script outside of the Makefile for
publishing. Or override CODEMAPFLAGS.
- Adopted implicit prefixing, -q, etc. This simplifies some of the more
complicated csv.py invocations (make summary, make funcs, etc).
See make help for a full list of commands.
This replaces the previous fallback-to-what's-available behavior with
explicit flags:
- --tile-code - Tile based on code size (the default)
- --tile-stack - Tile based on stack limits
- --tile-frames - Tile based on stack frames
- --tile-ctx - Tile based on function context
- --tile-1 - Tile functions evenly
This has the benefit of 1. being easier to toggle, 2. being explicit,
and 3. allowing code/stack/ctx in punescapes (titles, labels, etc).
There is an interesting question if --no-stack should be implicit, since
showing two stack treemaps may be confusing, but I think that's trying
to be too clever. Instead I just added the -S/--no-stack shortform to
make it easier to toggle.
Also updated ctx.py's description string. Probably need to check what
else is out of date in other scripts as well.
Now that I know my way around the weirdness that is Python's class
scope, this just required another function indirection to capture the
class-level dicts correctly.
I was considering using the __subclasses__ trick, but it seems like that
would actually be more complicated here.
This is limited to dbgle32.py, dbgleb128.py, and dbgtag.py for now.
This more closely matches how littlefs behaves, in that we read a
bounded number of bytes before leb128 decoding. This minimizes bugs
related to leb128 overflow and avoids reading inherently undecodable
data.
The previous unbounded behavior is still available with -w0.
Note this gives dbgle32.py much more flexibility in that it can now
decode other integer widths. Uh, ignore the name for now. At least it's
self documenting that the default is 32-bits...
---
Also fixed a bug in fromleb128 where size was reported incorrectly on
offset + truncated leb128.
Do you see the O(n^2) behavior in this loop?
j = 0
while j < len(data):
word, d = fromleb(data[j:])
j += d
The slice, data[j:], creates a O(n) copy every iteration of the loop.
A bit tricky. Or at least I found it tricky to notice. Maybe because
array indexing being cheap is baked into my brain...
Long story short, this repeated slicing resulted in O(n^2) behavior in
Rbyd.fetch and probably some other functions. Even though we don't care
_too_ much about performance in these scripts, having Rbyd.fetch run in
O(n^2) isn't great.
Tweaking all from* functions to take an optional index solves this, at
least on paper.
---
In practice I didn't actually find any measurable performance gain. I
guess array slicing in Python is optimized enough that the constant
factor takes over?
(Maybe it's being helped by us limiting Rbyd.fetch to block_size in most
scripts? I haven't tested NAND block sizes yet...)
Still, it's good to at least know this isn't a bottleneck.
These mimic dbgtag.py, but provide debugging for the lower-level integer
primitives in littlefs:
$ ./scripts/dbgleb128.py -x 2a 80 80 a8 01
2a 42
80 80 a8 01 2752512
$ ./scripts/dbgle32.py -x 2a 00 00 00 00 00 2a 00
2a 00 00 00 42
00 00 2a 00 2752512
dbgleb128.py is probably going to be more useful, but I figured we might
as well include both for completeness. Though dbgle32.py is begging to
be generalized.
This just gives dbgtag.py a few more bells and whistles that may be
useful:
- Can now parse multiple tags from hex:
$ ./scripts/dbgtag.py -x 71 01 01 01 12 02 02 02
71 01 01 01 altrgt 0x101 w1 -1
12 02 02 02 shrubdir w2 2
Note this _does_ skip attached data, which risks some confusion but
not skipping attached data will probably end up printing a bunch of
garbage for most use cases:
$ ./scripts/dbgtag.py -x 01 01 01 04 02 02 02 02 03 03 03 03
01 01 01 04 gdelta 0x01 w1 4
03 03 03 03 struct 0x03 w3 3
- Included hex in output. This is helpful for learning about the tag
encoding and also helps identify tags when parsing multiple tags.
I considered also included offsets, which might help with
understanding attached data, but decided it would be too noisy. At
some point you should probably jump to dbgrbyd.py anyways...
- Added -i/--input to read tags from a file. This is roughly the same as
-x/--hex, but allows piping from other scripts:
$ ./scripts/dbgcat.py disk -b4096 0 -n4,8 | ./scripts/dbgtag.py -i-
80 03 00 08 magic 8
Note this reads the entire file in before processing. We'd need to fit
everything into RAM anyways to figure out padding.
This matches the behavior of paths and helps figure out which string is
associated with which crc32c/parity when checksumming multiple strings:
$ ./scripts/crc32c.py -s hi hello
f59dd9c2 hi
9a71bb4c hello
It also might help clear up confusion if someone forgets to quote a
string with spaces inside it.
- Added TreeArt __bool__ and __len__.
This was causing a crash in _treeartfrommtreertree when rtree was
empty.
The code was not updated in the set -> TreeArt class transition, and
went unnoticed because it's unlikely to be hit unless the filesystem
is corrupt.
Fortunately(?) realtime rendering creates a bunch of transiently
corrupt filesystem images.
- Tweaked lookupleaf to not include mroots in their own paths.
This matches the behavior of leaf mdirs, and is intentionally
different from btree's lookupleaf which needs to lookup the leaf rattr
to terminate.
- Tweaked leaves to not remove the last path entry if it is an mdir.
This hid the previous lookupleaf inconsistency. We only remove the
last rbyd from the path because it is redundant, and for mdirs/mroots
it should never be redundant.
I ended up just replacing the corrupt check with an explicit check
that the rbyd is redundant. This should be more precise and avoid
issues like this in the future.
Also adopted explicit redundant checks in Btree.leaves and
Lfs.File.leaves.
Two new tricks:
1. Hide the cursor while redrawing the ring buffer.
2. Build up the entire redraw in RAM first, and render everything in a
single write call.
These _mostly_ get rid of the cursor flickering issues in rapidly
updating scripts.
This allows -w to provide a shortform flag for both --wear and
--block-cycles, depending on if you include a cycles argument:
- -w => --wear
- -w100 => --block-cycles=100
I was originally hesitant to add this since it's inconsistent from
--read/--prog/--erase, which can't have shortforms due to flag
conflicts, but --wear is probably a special enough case.
Christopher Haster