The *_is* functions for testing for bit flags have proven useful, maybe
mostly due to C's bad bitwise operator precedence, but the
*_set*/*_clear* functions just add extra code without much benefit.
Relying on explicit bitwise operations also lets us use the |=/&= shortcut
operators, which are nice.
Get it? Because they're small trees!
Joking aside, having a new term for these helps structure and describe
the filesystem at a high-level without needing to say "inlined trees"
all the time.
Shrub trees are small rbyd trees inlined directly in a file's mdir.
The main improvement is moving the special inlined-file compaction logic
up into lfsr_mdir_compact__. We only need this logic for files stored in
mdirs, and thanks to its recursive nature, we weren't getting any
benefit from handling this at a lower level anyways.
This is a nice logical restructuring that probably saves a bit of code
cost in the end.
Another significant improvement is moving the staging copy of the
inlined tree's state up into the file struct itself. This solves the
problem of needed N copies of temporary inlined state when you have N
open files.
It also provides a central place to stage changes when compacting
inlined trees, which happens across several different places in the mdir
commit logic. Though some may see this as more a hack than a feature.
Also note-worthy, but minor: these changes required an additional
opened-mdir linked-list to know when the mdir is a file and may contain
an inlined tree.
Ran into an interesting macro-related bug. Turns out the way we are
doing implicit prefixing in TAG/ATTR macros sort of breaks how C macros
work a bit. The following does not compile:
lfsr_mdir_commit(lfs, &file->m.mdir, LFSR_ATTRS(
LFSR_ATTR(file->m.mdir.mid, DEFER, 0, DEFER(
(lfsr_rbyd_t*)&file->inlined,
LFSR_ATTR(file->buffer_pos,
DEFERRED(INLINED), +file->buffer_size, BUF(
file->buffer, file->buffer_size))))));
Or to distill it down, this does not compile:
#define LFSR_ATTR(_data) (LFSR_##_data)
#define LFSR_DEFER(_data) (LFSR_##_data)
#define LFSR_DATA(_data) (_data)
int a = LFSR_ATTR(DEFER(ATTR(DATA(1))));
But this does:
#define LFSR_ATTR(_data) (_data)
#define LFSR_DEFER(_data) (_data)
#define LFSR_DATA(_data) (_data)
int a = LFSR_ATTR(LFSR_DEFER(LFSR_ATTR(LFSR_DATA(1))));
Why? Well it turns out the whole way nested C macro's work is a big
hack.
A very reasonable design decision in C is to disallow recursive macro
expansions. Unlike C++, we don't want our preprocessor to suddenly stack
overflow. This rule is enforced by stopping macro expansion when a macro
contains itself. For example:
#define A() B()
#define B() A()
A()
Expands to:
A()
-> B()
-> A() (stops, probably erroring with 'A' undeclared)
But it _is_ common to want to recursively expand macro arguments. Macros
are a part of C's syntax after all, and users usually expect
expressions, such as arguments, to be context-free:
#define A(x) (x) + 1
A(A(A(A(A(0)))))
Naively this would expand to:
A(A(A(A(A(0)))))
-> (A(A(A(A(0))))) + 1 (stops)
The big hack that makes this work in C's preprocessor is the "Argument
prescan". Instead of expanding the "called" macro first, we expand any macro
inside our argument list, _then_ expand the "called" macro, and _then_
expand any new macros produced as a result of the expansion again just
for good measure.
So the above actually expands to:
A(A(A(A(A(0)))))
-> A(A(A(A((0) + 1))))
-> A(A(A(((0) + 1) + 1)))
-> A(A((((0) + 1) + 1) + 1))
-> A(((((0) + 1) + 1) + 1) + 1)
-> (((((0) + 1) + 1) + 1) + 1) + 1
This is still recursive actually! But the recursion is limited to the
actual length of the source code, so the developers likely thought this
was a reasonable tradeoff.
But what does this mean for our implicit prefixing?
#define P_A(x) P_##x
#define P_B(x) P_##x
#define P_C(x) (x)
P_A(B(A(C(0))))
None of A, B, C are in scope without prefixes, so they get expanded
after the "called" macro's expansion:
P_A(B(A(C)))
-> P_B(A(C(0)))
-> P_A(C(0)) (stops)
But this breaks when we hit the nested P_A macro.
---
For now I've gone with the temporary, and extra hacky, solution of
introducing a second LFSR_ATTR_ macro. This nesting of ATTR macros only
happens because of shrubs, and only ever goes 2 layers deep.
In the future maybe we should move away from implicit prefixing. They
have a few rough corners and may be a bit confusing for anyone new to
the code.
Currently limited to inlined files and only simpler truncate-writes.
But still this lets us test file creation/deletion.
This is also enough logic to make it clear that, even though we have
some powerful high-level primitives, mapping file operations onto these
is still going to be non-trivial.
Now, instead of storing a single contiguous block of config data, config
is stored as tagged metadata like any other attribute.
This allows more flexibility towards adding/removing config in the
future, without cluttering up the config with deprecated entries (see
ATA's "IDENTIFY DEVICE" response).
Most of the config entries are single leb128 limits on various integer
types, with the exception of the magic string and version (major/minor
pair).
---
Note this also includes some semantic changes to the config:
- Limits are stored as size-1. This avoid issues with integer overflow
at extreme ranges.
This was also adopted for block size (block limit) and block count
(disk limit). This deviation between on-disk config and user-facing
config risks confusion, but allows the potential for the full 2^31 range
for these values.
- The default cksum type, crc32c, has been changed to 0.
Originally this was 2 to allow the type to map to the crc width for
crc8, crc16, crc32c, crc64, etc. But dropping this idea and numbering
checksums as they are implemented simplifies things.
May come back to this.
- Storing these configs as attributes opens up of the option of on-disk
defaults when configs are missing.
I'm being a bit conservative with this one, as it's not clear to me if
we should prefer default configs (less code/storage, risk of untested
config parsing) or prefer explicit on-disk configs.
Currently the following have defaults since they seem the most obvious
to me:
- cksum type => defaults to crc32c
- redund type => defaults to parity (TODO, should this default to
no redund?)
- utag_limit => defaults to 0x7f (no special tag decoding)
- uattr_limit => defaults to block_limit (implicit)
- mbits -> mleaf_bits
- mlimit -> mleaf_limit
- mweight -> mleaf_weight
- lfsr_mridmask -> lfsr_midrmask
- lfsr_mbidmask -> lfsr_midbmask
This is a bit tricky to name, since we want to clarify it's not the
mtree limit and not the mdir's actual rbyd weight. But this also risks
confusing around the difference between mdirs/mleaves (mdirs are
mtree's leaves).
- Fixed LFSR_GRM_DSIZE upper bound, since our mids now fit in a single
leb128.
- Renamed pgrm -> ggrm. To be honest I don't have a great name for this
variable.
Adopted lfsr_rid/bid/mid/did_t where appropriate. This includes using
lfsr_rid_t for tag/rbyd weights. Although I am using lfsr_srid_t for
rbyd weights now, since it both captures the use of the sign bit and
reduces the number of casts a bit in the code.
I learned recently Zig has any-bit integers (e.g. uint31_t), and I'm
realizing how nice it would be to have those in this codebase.
Also tried to use lfs_size_t/lfs_off_t more correctly. In Linux/BSD,
only off_t is used for file-size-related operations and is usually much
larger than size_t. These were used interchangably in littlefs and their
original meaning kind of fell by the wayside. Getting their use right
will be important if littlefs ever supports different integer widths.
This reverts the big hack of treating the lfsr_dir_t as an mdir array in
lfsr_mdir_commit in an effort to deduplicate the bookmark/pos mid
updates.
It worked, but lets be honest, it was a big hack and probably not very
maintainable. It made other opened-mdir updates, such as propagation of
unerases more complex, and is made the code a bit unreadable.
We also don't really need a full mdir for the dir's bookmark, since
rewinds really aren't that common, and a single mtree lookup in that
case gets the job done. Removing the bookmark mdir (though we still need
the bookmark mid to adjust the dir pos correctly) saves 24 bytes from
every lfsr_dir_t.
It would be nice to deduplicate some of the mid logic here, but that's
been difficult because of mid-related side-effects, such as updating the
mdir's pos. There may be room for improvement here.
---
This looks pretty bad, with the additional loop over the attr-list to
update just the dir's bookmark, but it's really not that bad when
compiled, and probably worth the code readability:
code stack structs
before: 20958 1744 864
after: 21052 (+0.4%) 1744 (+0.0%) 840 (-2.9%)
This is a tricky nuance of how rbyd's erased state interacts with
possible errors during commits.
- If an rbyd passes its ecksum during rbyd-fetch, it's erased and we can
write to it.
- If an rbyd is committed to successfully and still has erased space
remaining, it's erased and we can write to it.
- But if we fail to commit to the rbyd, we can't be sure the trailing
data is still erased. It most likely isn't, and we would need to fetch
again to check the ecksum. And since errors are exceptional here, we
might as well just mark any failed commits as unerased, triggering a
compaction on the next write to the rbyd.
To make things more annoying, changing state in all error routes is
tricky to get right, and trickier to test. To keep this relatively
simple and robust, all rbyd/btree/mdir operations mark the original copy
as unerased until the commit succeeds, and then clears the unerased
state. This fits in well with how we make copies of the rbyd/btree/mdir
structs in the relevant functions.
Note this needs to affect _all_ copies of the rbyd, including any opened
mdirs, mroots, etc. This will probably still lead to some bugs in the
future...
- Added lfsr_mdir_lookupnext, for iteration through only a single mid.
This is useful for MOVE attributes.
- Renamed LFSR_MDIR_MROOTANCHOR -> LFSR_MROOTANCHOR.
- Renamed functions that operate on mdir blocks lfsr_mdir_* ->
lfsr_mblocks_*.
- Reordered arguments in lfsr_mdir_fetch.
- Renamed mrid_bits/mbid_weight -> mbits/mweight.
This adopts a previously discarded idea for compressed mids with a few
tweaks to avoiding decoding the bid/rid portions as much as possible.
The idea of compressed mids is to shove both the mid bid and mid rid
into a single integer, saving RAM and potentially helping filesystem
integration where a unique per-file integer is useful.
Unfortunately this has proven tricky. littlefs fundamentally needs two
ids, one "bid" to lookup which mdir our entry resides on, and one "rid"
to lookup the entry in the mdir. It's tempting to use two half-sized
integers (16-bit for example), but this risks surprising limitations
around the number of files when blocks are either really large or
really small.
Optimally, we'd limit the number of bits reserved for the rid to the
upper bound of number of rids that can fit in a single mdir. This would
allows for more bids when the block size is small, and more rids when
the block size is large. This should roughly approximate the limits of
a per-file integer.
With a bit of math we can estimate the upper bound to be <=block_size/16
with our current compaction strategy.
This idea was previously discarded due to the overhead of extracting the
bids/rids when we need them, but the RAM savings and file-to-integer
mapping is too useful to give up. When it became clear half-width
integers wasn't really going to work, compressed mids became the new
plan:
0bbbbbbb bbbbbbbb bbbbbbbb rrrrrrrr
^'-----------+-----------' '---+--'
'------------|-----------------|---- sign-bit, reserved for driver
| '---- nlog2(bs/16) bits for rid
| (8-bits for 4KiB blocks)
'---------------------- remaining bits for bid
(23-bits for 4KiB blocks)
To reduce the overhead of encoding/decode bids/rids a few extra features
were added to the internal mdir APIs:
1. The mtree has been changed to store mids directly. Giving each mdir
the upper bound as a weight. This allows direct lookup of mids
without any sort of bid decoding, though does bake the upper bound
estimate into the metadata of the filesystem, which isn't the
cleanest design, but if it works it works.
On the plus side, with this upper bound baked in to the filesystems,
GRMs can be encoded in a single leb128, which is nice. This may have
other savings if we ever store mids anywhere else in the filesystem.
2. rids are now mid relative in lfsr_mdir_lookup when non-negative. This
is implemented with a simple condition that is hopefully optimized
out when inlined, though there may be some room for improvement here.
3. rids are now mid relative in lfsr_mdir_commit. This was a bit tricky,
but we can leverage the existing mechanisms for bid-relative rids
used in the btree implementation.
The above changes make it so you can pass the mid around directly for
most of the mdir functions, hopefully reducing the mid decoding
overhead. This savings should only grow as more high-level filesystem
APIs are added.
Here is the resulting code/RAM changes for this entire change (from
before we adopted the mroot bit):
code stack structs
before: 20590 1784 908
after: 20890 (+1.4%) 1744 (-2.3%) 864 (-5.1%)
More on this when explaining compressed mids, but basically the idea is
instead of just storing all mdirs in our mtree as single element
entries, store each mdir in as a weighted entry, where the weight is a
known upper bound on the possible number of mid entries in a single
mdir.
With the current mid representation, this just complicates things
without much benefits. But with compressed mids it allows us to lookup
mdirs with the mid directly, and avoid decoding the bid from the mid in
some cases.
The mid-per-mdir upper bound is derived from the block size. We know:
1. Each tag needs <=2 alts+null with our current compaction strategy
2. Each tag/alt encodes to a minimum of 4 bytes
This gives us ~4*4 or ~16 bytes per mid at minimum. If we cram an mdir
with the smallest possible mids, this gives us at most ~block_size/16
mids in a single mdir before the mdir runs out of space.
Note we can't assume ~1/2 block utilization here, as an mdir may
temporarily fill with more mids before compaction occurs.
This is an intermediate commit as a part of a tangent into compressed
mids.
The idea here, is instead of using bid=-1 as a special value for mroots,
use only the top bit to indicate mroots. This allows you to compare
against the grm/other uninlined mids by masking instead of signed
comparison.
This is valuable for compressed mids since extracting bids relies on
knowledge of the block size, and becomes quite a bit more expensive.
mroot bid mroot cmp
before: 0xffffffff lfs_smax32(a, 0) == lfs_smax32(b, 0)
after: 0x80000000 (a & 0x7fffffff) == (b & 0x7fffffff)
The implementation here is a bit clumsy. I think GCC may be not that
great at optimizing out copies of structs being passed around via
inlined functions. But this is only a proof-of-concept.
The possibility of 16-bit mbid/mrids being a problematic limit is too
high for me to be able to confidently move forward with this internal
encoding.
Consider a filesystem with small blocks and/or large amounts of metadata
per file. If a single file almost fills up an mdir, it risks an
effective limit on the filesystem of 2^16 files. Not a deal-breaker, but
certainly a surprising limit on a supposed "32-bit" filesystem.
We can add more granular integer-limit configurations, but with simple
configurations, the option to increase the integer-limit filesystem-wide
to 64-bits would cost more RAM than just increasing the mbid/mrids
limits.
Though this is always up for reconsideration in the future.
code stack
before: 20762 1720
after: 20590 (-0.8%) 1784 (+3.7%)
It's interesting to not the code/RAM tradeoff here. RAM sees a
significant hit, but code improves, likely because of better instruction
sequences for 32-bit operations (this is targeting ARM thumb,
32-bit MCUs).
Though the code savings likely varies widely across instruction sets,
and I would guess swings negative on 16/8-bit MCUs.
The idea here: Instead of having unique functionality for each
individual btree operation (push/set/pop/split), we treat btrees sort of
like rbyds, with a single commit entry point that operates on attr-lists.
This adds code cost, due to needing to parse the attr-list for properties
that can affect inlined btrees (tag changes mostly), but, in theory, comes
with some advantages:
1. A single btree commit entry point with all of the inlined/uninlining
logic should offer better chances for code deduplication, vs
spreading this logic out in each btree operation.
2. Higher-levels should know what the current weight of the branch is,
so we may be able to avoid the implicit math needed to calculate
deltas.
3. Higher-levels have more knowledge about the state of the btree in
general, so there may be other shortcuts. The mtree, for example,
only operates on weight=1 entries, which greatly simplifies a lot of
the related math.
Note that btrees still have strict limits in what's possible in an
attr-list. Btree operations can't cross leaf-rbyd boundaries for
example.
---
A notable omission in this change is the loss of reinlining btrees.
This wase dropped for a couple reasons. It may be worth adding back at a
later time, maybe after we actually have files implemented, but for now
does not seem worth it:
1. Reinlining adds code cost. Reinlining is more complex than you might
expect because we only reinline on compaction. And because we compact
before playing out our attr-list, we need to know if a commit makes
the btree inlinable before committing to the btree.
This is still doable with our attr-lists. We already derive the
change in tags, since we need this to know when to uninline. But it
adds a kind of complex bailing out of btree commits.
2. The benefits of reinlining may not be that great. In most systems, a
tree that is uninlined once is likely to be uninlined again. It's
only if there is a bigger state change in a system that it makes
sense to reinline.
Though, to be fair, waiting for compaction to reinline handled this
quite well. Only reinlining when all erased storage is used up...
3. Thanks to our roots did entry, our mtree can never reinline.
It would be nice to change this, but this would require explicit
handling in lfsr_mdir_commit. Future work?
4. Files are another can of worms, with more complex interactions with
inlinability thanks to (at least on paper right now) always having
inlined data even when uninlined.
If reinlining is valuable for files this can change during that work.
5. Even if files never support reinlinability, truncating files (via
either lfsr_file_truncate or LFSR_O_TRUNC) should give the file a
blank slate, effectively reinlining the file in that case.
---
The current implementation also changes the attr-list to be mutable so
we can adjust attr-list based on the current btree node. This is a
temporary hack! We should add the appropriate functionality to our rbyd
utilities to revert this eventually.
- lfsr_mid_cmp no longer uses a union. This was undefined behavior and
the lfsr_mid_t type isn't word aligned, so this could break pretty badly
on machine/compiler change.
Also dropped ordering based on endianness, since we need to marshal
these into an int for the comparison anyways.
- Changed lfsr_mdir_cmp to use min/max functions as part of the
comparison. The result is also "ordered" now, though the ordering
is nonsensical. I guess the mrootanchor is less than all other mdirs?
Also considered only comparing a single min/max block, since it would
be an error for mdirs to share blocks, but note we rely on
lfsr_mdir_cmp to check for relocations in lfsr_mdir_commit. These
relocations can end up being partial in the case of bad block
detection.
Generally the more creative you get with C macros, the more
unmaintainable your codebase becomes, but in this case I think a small
bit of macro sugar for the attribute lists in littlefs goes a long way
for making the internals flexible and readable.
Attribute lists generally look like this:
LFSR_ATTRS(
LFSR_ATTR(id, TAG, delta, DATA(data)),
LFSR_ATTR(id, TAG, delta, DATA(data)),
...
LFSR_ATTR(id, TAG, delta, DATA(data)))
Which more-or-less gets expanded to this:
((const lfsr_attr_t[]){
((lfsr_attr_t){id, LFSR_TAG_TAG, delta, LFSR_DATA_DATA(data)}),
((lfsr_attr_t){id, LFSR_TAG_TAG, delta, LFSR_DATA_DATA(data)}),
...
((lfsr_attr_t){id, LFSR_TAG_TAG, delta, LFSR_DATA_DATA(data)}),}),
attr_count
Note the use of preprocessor concatenation to put the TAG and DATA
identifiers in their respective namespaces. These can end up invoking
other macros, which allows attrs to be rather extensible.
Previously there were also LFSR_ATTR_ (note the trailing underscore)
macros to allow passing of variable tags/datas. This is replaced with
redundant macros which sort of "unwrap" themselves as a part of macro
expansion. This avoids a bunch of duplicate macro definitions.
#define LFSR_TAG_TAG(tag) (tag)
#define LFSR_DATA_DATA(data) (data)
So:
LFSR_ATTR(id, TAG(tag), delta, DATA(data))
Becomes:
((lfsr_attr_t){id, LFSR_TAG_TAG(tag), delta, LFSR_DATA_DATA(data)})
Becomes:
((lfsr_attr_t){id, tag, delta, data})
This checksum is used to keep track of if we have erased, and not yet
touched, the unused bytes trailing our current commit in the rbyd.
The working theory is that if any prog attempt is made, it will, most
likely, change the checksum of the contents, allowing littlefs to
determine if trailing erased-state is safe to use, even under powerloss.
littlefs can also perturb future data by a single bit, to force this
checksum to always be invalidated during normal operation.
The original name, "forward erased-state checksums (fcksum)", came from the
idea that the checksum "looks forward" into the next commit.
But after using them for a bit, I think the name is unnecessarily
confusing. It, uh, also looks a lot like a swear word. I think
shortening the name to just "erased-state checksums (ecksum)", even
though the previous name is already in use in a release, is reasonable.
---
It's probably hard to believe but the name change from fcrc -> ecrc
really was unrelated to the crc -> cksum change. But boy is it
convenient for avoiding an awkward name. A lot of these name changes
involved sed scripts, so I didn't notice how awkward fcksum would be to
use until writing this commit message.
The reason for this is to move away from the idea that littlefs is
strictly bound to CRCs and make the code more welcoming to other
checksum types, such as SHA256, etc.
Of course, changing the name doesn't really do anything. littlefs
actually _is_ strictly bound to CRCs in a couple ways that other
filesystems aren't. These would need to have workarounds for other
checksum types:
- We leverage the parity-preserving nature of (some) CRCs to not have
to also calculate the parity of metadata in rbyd commits.
- We leverage the linearity of CRCs to retroactively flip the
perturb bit in the cksum tag without needing to recalculate the
checksum. Though the fact we need to do this is because of how we
use parity above, so this may just not be needed for non-CRC
checksums.
- The plans for global-CRCs (not yet implemented) rely heavily on the
mathematical properties of CRC polynomials. This doesn't mean
global-CRCs can't work with other checksums, you would just need to
find a different type of polynomial.
For a couple reasons:
1. Organizing the overlaps this way avoid potential undefined behavior.
It turns out C does define the overlap the "initial sequence" of
union members, as long as the types are the same. But when we
overlapped the block with the size/tag fields in lfsr_btree_t, it was
probably undefined behavior.
At the very least, it would introduce a need for quite a bit of
preprocessing to make it work with different integer sizes and
redundancy levels.
2. Overlapping the blocks at the end of the rbyd struct means our block
array is natural ordered such that the first block is the "active"
block, i.e. the block with the most recent revision count that passes
checksums.
This has been useful as a debugging tool, so I would like to continue
the pattern. It is possible to mostly preserve this order with the
previous method by intentional reversing the block array when
logging or writing to disk, but it's a bit cumbersome.
2. It's unlikely we'll be able to use readonly variants of the rbyd/mdir
structs for RAM savings. Unfortunately C makes this too cumbersome.
Though if we do this should be revisited.
Here are the new overlaps. Note it's no longer possible to truncate the
types when readonly. If readonly struct are useful this will need to be
revisited again:
lfsr_rbyd_t lfsr_btree_t lfsr_mdir_t
8b 8b 8b 8b
.----+----+----+----.
8b 8b 8b 8b 8b 8b 8b 8b | mid.bid | mid.rid |
.----+----+----+----. .----+----+----+----. |----+----+----+----|
| weight |.>| weight | | weight |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| trunk | | tag | size | | trunk |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| off | | inlined data | | off |
|----+----+----+----| | | | |----+----+----+----|
| crc | | v | | crc |
|----+----+----+----| | | |----+----+----+----|
| block |..| |.>| blocks |
'----+----+----+----' '----+----+----+----' | |
| |
'----+----+----+----'
This turned out to be tricky.
At littlefs's core, we have the lfsr_rbyd_t struct. It is really
important this is as small as possible since littlefs creates many rbyd
copies in order to track state of metadata on disk.
Wrapping rbyd, we have the lfsr_btree_t struct, which can alternatively
contain a single inlined entry, accomplished by overlapping the width
field in both cases. And the lfsr_mdir_t struct, which tracks any redundant
blocks, and would be nice if the blocks lined up as neighbors so all blocks
involved in the mdir could be passed around as an array. Both of these
wrappers attempt to overlap fields of the lfsr_rbyd_t struct, which presents
a bit of a problem.
The solution here is to put the rbyd block field at the beginning of the
lfsr_rbyd_t struct, and use exactly 32-bits of padding in lfsr_btree_t
to overlap the width field even though it is not at the beginning of the
struct. To avoid inflating the lfsr_btree_t size, we sneak the inlined
size and tag into the overlapping padding. This will need special
handling if the size of these fields change, but saves a decent amount
of RAM:
lfsr_rbyd_t lfsr_btree_t lfsr_mdir_t
8b 8b 8b 8b
.----+----+----+----.
| mid.bid | mid.rid |
|----+----+----+----|
8b 8b 8b 8b 8b 8b 8b 8b | blocks |
.----+----+----+----. .----+----+----+----. | |
| block |..| tag |size|padd|.>| |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| weight |.>| weight | | weight |
|----+----+----+----| |----+----+----+----| |----+----+----+----|
| trunk | | inlined data | | trunk |
|----+----+----+----| | | | |----+----+----+----|
| off | | v | | off |
|----+----+----+----| | | |----+----+----+----|
| crc | | | | crc |
'----+----+----+----' '----+----+----+----' '----+----+----+----'
Also tried to reduce the amount of mdir usage in lfsr_mdir_commit by
better using only the arrays of relevant mdir blocks, to limited success.
- Updated LFSR_BTREE_INLINESIZE to properly include the overhead for
mdir pointers, which need 2 block addresses instead of 1. This adds
4 bytes to the lfsr_btree_t struct.
- Changed code that marks rbyds as "needing compaction" to use -1
instead of block_size. This can use a cheaper constant and helps
debugging.
- Changed the mid representation of root to 0.0 from ?.-1. The mid 0.0
is always reserved for the roots dstart, so it shouldn't be used for
any actual file. This disambiguates root vs special metadata mids and
is a step towards making mids unsigned.
It also saves a tiny bit of code since 0 comparisons are generally
cheaper and we can leverage the order-preserving conversion of mid
to an integer.
This didn't really work out as well as I had hoped. There were a few
ideas on how to encode the bid/rid tuple without sacrificing the
(currently 31-bit) integer limit, but these just introduced too much
complexity.
Ideas:
1. In theory, as the mdirs increase in size, the quantity of mdirs needed
for a given number of files decreases. If we say the number of files
fits in an integer of a given size, than we can model the mapping to
mdirs and rids roughly as the number of bits in that integer split
between the two.
Since the block_size is known, the we can find a rather conservative,
yet useful, estimate of the upper bound of rids, which ends up
being ~16 bytes ((2 alts + 1 null + 1 tag) * 4 bytes).
And since our btrees are perfectly balanced, this encoding should only
waste 1 or 2 bits due to rounding to rounding and sign encoding for
special values.
bbbbbbbb bbbbbbbb bbbbbbbr rrrrrrrr
'-----------+-----------''----+---'
| '-- log2(block_size/32)-bit rid
'-------------------- remaining-bit bid
Unfortunately, while this works ok on paper, and maximize the use of
the bits we have available for the mid, the implementation ended up
awkward and difficult to use.
We need to either calculate the relatively complciated log2 of the
block_size on the fly, or cache the value, and use it to shift the
mid around to extract the bid/rid when needed.
Unfortunately, perhaps due to the it being easy to use the bid/rid
directly, we use and mutate the bid/rid quite a bit. We mutate when
updating the mdirs, when decoding grms, when seeking mdirs, etc. If
anything, updating the mid in total is rarer than updating the
bid/rid component in complicated situations.
Note to mention this required access to the lfs config to even begin
decoding, complicating the API and making the result less efficient.
Initial (unoptimized, and not even tested) code size showed ~+800
bytes. So I decided to scrap this.
Maybe it will be worth investigating dynamic rid sizes later, to
increase the possible mtree size for a given mid width. Not sure.
2. Probably one of the worst ideas I've had so far, but it would solve
the mid encoding problem, is to use some form a floating point to
encode the bid/rid pair:
.----------.
v .+-.
bbbbbbbb bbbbbbbb bbbrrrrr rrrrssss
'-----------+-------''----+---''-+'
| | '-- rid bits
| '--------- variable rid
'----------------------- variable bid
An even worse idea would be to use IEEE floating point here. Yes it
would work, and probably work annoyingly well, but we it risk
bringing in a lot of standard conforming backbending that we really
don't care about.
The idea here is to sacrifice some bits to encode the ratio of rid
bits to bid bits. The value of this over the using the block_size is
that we can decode the bid and rid using all of the bits in the
integer alone. Avoiding memory access (and worse debugging) to load
any external constants.
As a plus, all mids in the system would have the same exponent,
simplifying comparisons and other operations.
But this is just trying to solve complexity by adding more
complexity, so I'm not even going to try implementing it.
Still, it's an interesting idea...
In the end I've gone with the KISS implementation. Use half-width
integers, in this case uint16s, for both the bid and rid:
bbbbbbbb bbbbbbbb rrrrrrrr rrrrrrrr
'-------+-------' '-------+-------'
| '-- 16-bit rid
'-------------------- 16-bit bid
This suffers from weakened limits around the number of rids in a block
and number of mdirs in the mtree, which is unfortunate. Still it is
probably worth the tradeoff for the RAM savings and encoding simplicity.
If the mdir is reasonably sized, this does probably approach a decent
distribution of rids and bids in 32-bits. But for outlier cases with
very small and very large mdirs, it risks premature out of bounds
errors.
To protect against mtree errors, we will probably need an additional
configuration option in the form of an mdir limit. Conveniently this
would also provide a way to enforce 2-block mode.
rid errors, on the other hand, depend on block_size/32, so we may not
need another configuration option and can rely on the block_size
to determine if the rids can overflow.
This is probably worth revisiting in the future. Fortunately, with
mdir_limit and block_size configuration options, it should be possible
to increase these limits in the future if this mid bid/rid design
changes.
code stack
before: 22126 2136
after: 22326 (+0.9%) 2088 (-2.2%)
This code size increase was unexpected. Maybe non-32-bit-aligned integers
cost more to load in thumb? Unsure.
The main intention here was to make the tracking of opened mdirs,
mostly opened lfsr_dir_t structs, simpler and more resilient to weird
corner cases. I'm not entirely sure this was successful.
The main changes:
- lfsr_dir_t now contains a full mdir for the dstart entry.
This makes it so that dstarts are not a special case when it comes
to mdir updates, though the fact that directories have 2 mdirs is
still an awkward case on its own.
I considered using two entries in the opened linked-list for this, but
it wouldn't have worked out that well. Both entries need to update the
directory position, so it would have required a third file type. We
would also have needed to make sure removed mdirs mark both mdirs as
removed, otherwise the position mdir would move around arbitrary into
possibly erronous values.
Instead the current solution treats the directory mdirs as a small
array of 2 mdirs, which is as hacky as it is hacky, but does get the
job done with little code duplication.
- Directory positions are updated a bit more intellegently.
Instead of checking if in range before updating, which requires access
to both mdirs and duplicate mid/rid comparison logic, position is
updated without regard for the beginning of the directory, and
un-updated if it was actually out of range of the directory.
This means we only need to compare the mids/rids for each mdir once.
This changes make it so that lfsr_dir_rewind is much cheaper, and
doesn't even need to go to disk. Though I'm not sure it's worth the RAM
increase...
Expanding the lfsr_dir_t dstart entry to a full mdir does a lot for
making mdir updates more consistent, but increases the lfsr_dir_t size
from 52 bytes to 76 bytes (+46.2%).
When updating any opened mdirs to keep things in sync, we need to know
what rid the mdir is targeting in order to know which on-disk mdir it
should follow in the case of splits. Making this rid an actual member of
the mdir struct simplifies things.
This adds some RAM cost, though the plan is to merge the mid/rid into a
single integer, which requires this change and should actually save RAM
in the long run.
code stack
before: 22342
after: 22204 (-0.6%) 2144 (+1.1%)
It turned out our dir-read-idempotent test never created non-dstart
neighbors. This was a bit of a problem since we relied on dstart entries
to know when our dir read terminates. If we seek to an invalid position
(in theory undefined behavior, but easily possible with concurrent
modifications to the directory), we can end up reading an unrealted,
non-dstart entry, and incorrectly reporting that entry as in our current
dir.
This fix reintroduces the did into the lfsr_dir_t struct and uses the
did to determine end-of-dir. This adds some RAM cost, but is more
resilient to any seeks that overshoot the end of the directory.
Using did is also a stronger guarantee we will never accidentally report
unrelated entries as a part of the current directory.
This is entirely a pragmatic change, lfsr_mdir_commit already does
several hairy things with grm tags, decoding, fixing, reencoding, etc,
so it makes sense to move all the encoding logic into lfsr_mdir_commit.
This leads to a couple optimizations:
- We don't need to decode the grm to apply any last minute fixes.
- By allowing the grm arugment to be mutated (they are just sitting on
the stack anyways, we need a copy in case we back out of change due to
error), we can apply and save any grm fixes in the grm argument
itself.
This means we only need to fix the grm at most once, after any mtree
modifications.
Which in turn saves some code and stack cost:
code stack
before: 22930 2392
after: 22706 (-1.0%) 2344 (-2.0%)
Unfortunately the previous attempt to fix the dir seek system didn't
really work. Using a packed mid/rid integer for the offset is tempting,
but since mid/rid can change with any metadata id change in the
filesystem, dir tell offsets would become invalidated if you modified
files in unrelated directories, which isn't great and likely to catch
users by surprise.
This solution builds on the previous dir offset design, which tracks the
dstart-relative position independently from the current mid/rid in our
directory. To update this correctly when there are unrelated changes to
the filesystem, we need to know if metadata id changes are in the range
between our directories dstart and current mid/rid. This in turn means
we need to track our dstart. So our opened directories need three
separate pointers we need to update on every mdir commit:
dir->pos
|
.-------+-------.
a b c d e f g h i j k l m n o p
^ ^
| |
dir->dstart dir->mdir
This has quite a few moving parts, which I was hoping to avoid.
Fortunately we don't need a second mdir, so the RAM cost is pretty
small.
We can also drop dir->did, since the dstart mid/rid render it redundant,
which is interesting.
This is an attempt to fix issues with dir seeking in a filesystem
undergoing changes. The problem with the previous dstart-relative
position encoding is that if we deleted/created new entries outside of
our current directory, we didn't if they were inside or outside of the
current directory, so we couldn't always update our position correctly.
Instead of using a dstart-relative position, this solution crams both
the mid and rid into a single 31-bit integer. Things get a bit tight
here, so we use the current block_size as a heuristic for how many
possible rids we can ever have in a single mdir. The idea is the larger
the rid encoding needs to be, the smaller the mid encoding needs to be,
and we should, _roughly_, approach the same encoding limitation we would
have to dstart-relative position anyways.
Making some assumptions about the maximum possible number of rids in a
block gives us at most ~block_size/8 rids per mdir.
So for 4096 byte blocks (note the exact encoding is dynamic):
sbbbbbbb bbbbbbbb bbbbbbbr rrrrrrrr
^'-----------+----------''----+---'
'------------|----------------|----- sign bit (used for errors)
'----------------|----- 22-bit metadata bid
'----- 9-bit metadata rid
Note this introduced as new, significant limitation on the number of
total mdirs in the system. Normally I would be against this solution for
that reason, however if we adopt this encoding elsewhere in the system it
may improve some RAM cost and in general simplify things by being able to
store any mid in a single integer. More work needs to be done here...
This approach needs some fleshing out and has its own issues (the
offset returned by tell quickly becomes out of date if the filesystem
is modified, but is that really a problem?), but it improves over the
previous implementation by making tell always correct at that moment.
The idea here is to combine the current mtree size with the theoretical
upper bound on the number of directories in a single mdir, assuming our
block size, to give us a heuristic for did truncation that does not
require any extra state.
- Each directory needs 1 name tag, 1 did tag, and 1 dstart
- Each tag needs ~2 alts with our current compaction strategy
- Each tag/alt encodes to a minimum of 4 bytes
- We can also assume ~1/2 block utilization due to our split threshold
This gives us ~3*3*4*2 or ~72 bytes per directory at minimum, or
rounding down, ~block_size/32 directories per mdir.
This is a nice number because for common NOR flash geometry,
4096/32 = 128, so a filesystem with a single mdir encodes dids in a
single byte.
The biggest benefit though is being able to drop the mlimit state from
the lfs_t struct.
---
Unfortunately, this change revealed several bugs.
It turns out __builtin_clz in GCC is undefined at 0, which caused our
lfs_nlog2 function to return incorrect values at 1. This was causing
our dids to all collide when the mtree was inlined, which was resolved
by the linear scanning that resolves dids, but was severely limiting
what exactly our tests covered.
Now that this is fixed (with a simple if statement in lfs_nlog2,
lfs_nlog2 now always has defined behavior, even at 0), several bugs
needed fixing:
- We update the rid based on attrs in lfsr_mdir_commit before updating
the mdir. If we have multiple attrs this causes the assert on
rid-in-bounds to trigger incorrectly. Just removed that assert for now.
- We needed to adjust second grms if they are affected by the fixing
of the first grm.
- Directory position updates are incorrectly updated if an unrelated
weight change occurs before an opened directory, but is not a part of
that opened directory.
This is NOT fixed yet, the current implementation is just broken
enough that I've just ripped it out for now (it was causing the
read_with_rms test to fail because pos backed up into the "."/".."
entries).
This needs some thinking to fix.
Because of that last, unfixed bug, tests are not all passing at the
moment. To pass testing -DSEEK=0 is needed to disable the failing tests.
Only simple tests right now, but the theory is sound.
This mainly required the addition of the fancy in-device move attribute,
which copies all tags associated with an rid from one rbyd to another in
a single transaction.
This is a carryover from the previous littlefs implementation, though it
is easier to implement here since it is effectively a range query on the
rbyd tree, which trees are really good at. This was intentional.
Oh and I suppose this also required implementing lfsr_rename, which has
a few corner cases to watch out for.
It is nice that both lfsr_remove and lfsr_rename can rely on
lfsr_fs_fixgrm to finish all of the removes, which wasn't previously
reasonable due to the overhead of deorphaning.
Ugh. I overlooked a weird corner case in rename's behavior that requires
changes to the grm to support.
POSIX's rename, which lfsr_rename is trying to match, supports renaming
files over existing files, effectively removing the previous file during
the rename.
This is supported, even if the files are directories, but with the
additional requirement that the previous directory is empty (matching
the behavior of lfsr_remove).
This creates a weird situation for littlefs. In order to remove
directories in littlefs, we need to atomically remove both the dstart
entry that reserves the directory's did and the directories entry in its
parent. This is made possible by using the grm to mark one entry as
pending removed while removing the other.
But in order to rename atomically, we need to use the grm to mark the
source of the rename as removed while creating/replacing the destination
of the rename.
So we end up needing two grms simultaneously.
This is extra annoying because the niche case of renaming a directory
over another empty directory is the only case where we need two grms,
but this requirement almost doubles the grm size both in-ram and
reserved in every mdir, from 11 bytes to 21 bytes, and increases the
lfs_t size by 28 bytes.
---
Anyways, this commit extends the grm to support up to two pending removes.
Fortunately the implementation was simple since we already have a type
field that can be extended, and grm operations just needed to be
changed from if statements to for loops.
Recursive removes is proving more challenging to implement than I
expected. The problem with the previous approach is that it moved the
mid into a potentially non-sensical position with the expectation it
would be updated in lfsr_dir_read because the rid overflows the current
weight (since dropping mdirs always set the weight to zero).
But if an unrelated mdir commit followed that happened to touch that
nonsense mid, the mdir would incorrectly be updated to the previous
block, causing problems for the dir's read state.
---
The solution here is to toss all of that out and rely solely on directory
position updates, which are a bit simpler.
So in lfsr_dir_read, if our mid/rid is deleted, we perform a full
rewind+seek to the new position. This can be more costly, but since the
most common case, recursive removal, leaves us with all mid/rids < pos
deleted, it should only add a single mtree lookup per lfsr_dir_read.
Also added prototypes for dir seek/tell/rewind, since we're using
they're logic for this. Though these aren't yet tested. These are built
on the new function lfsr_mtree_seek which captures the common logic of
seek over multiple mdirs in the mtree efficiently, and skips unnecessary
rid lookups where possible.
"Recursion" here just refers to the ability to remove entries in a
directory while iterating over it. This is very useful when you just
want a directory gone, and can be extended to a "true" recursive remove
straightforwardly. This mainly tests that mid/rid updates in opened
mdirs are correct.
To make this work, we need to update opened dirs differently than files,
since opened dirs do not get marked as removed when its rid is removed
and contain an additional position in the dir that needs to be updated.
To keep track of the different types, littlefs now contains 2
linked-lists for opened mdirs. Maybe these should be correctly typed,
but by hiding the specific types behind an array of mdir linked-lists,
we can more efficiently iterate over both lists when necessary.
We should probably compare this approach to the type-tagged approach in
the previous littlefs implementation, but I think the idea of an array
of type-hidden linked-lists just didn't come to me then. There was also
a bit more room in the mdir structs to hide a 1-bit type field. The mdir
structs here are getting pretty squeezed since they are used everywhere.
In theory this is pretty much the same as lfsr_mkdir, but backwards.
The main work was making the interactions between removing mids/rids and
the grm correct. This ends up meaning we just need to update the grm on
any mid/rid update the same way we update the list of opened mdirs.
On the plus side, it turned out to be possible to deduplicate the mdir
uninlining route a bit, by adding range argument to lfsr_mdir_commit_
and changing the write of the newly uninlined mtree/mdir to marking
mtree as dirty and then joining the common path.
This lets us move the pre-commit round of grm updates into a single
location in lfsr_mdir_commit, removing and extra function definition and
the related state marshalling while also simplifying the control-flow.
This also raises the question, can more lfsr_mdir_commit be deduplicated
more? Uninlining is a infrequent operation we don't really need to
optimize for.
---
Testing lfsr_remove also found a bug related to incorrect propagation of
when the mroot becomes "unerased" (when rbyd overflows). This raises the
concern that we're not propagating unerased-states very rigorously, and
unexpected errors may not allow the filesystem to resume.
This has never been in a very good place for littlefs, but would be
worth improving in the future.
Instead of truncating to exactly 28-bits for nice leb128 alignment, we
now truncate to ~the number of metadata entries, which must be >= ~2x
the number dids since each did needs a dir entry and dstart entry.
This has the downside of needing to actually keep track of an estimate
of the number of metadata entries, which is made a bit difficult due to
integer overflow issues (we can have more than 2^32 metadata entries),
but has the upside of allowing a full 2^32 number of dids worst case.
This is really unlikely, but it's nice to not need another configuration
option to control the did limit.
Another option would be to scale the hashes based on the number dids,
which would be a more direct solution. Unfortunately determining the
number of dids during mount requires a O(m*log(m)) scan of each rbyd
to find either dir entries or dstart entries. This solution can easily
end up with an overestimate, but only needs to weight of each rbyd which
can be (and already is) found in O(m).
This implementation is in theory correct, but of course, being untested,
who knows?
Though this does come with remounting added to all of the directory
tests. This effectively tests that all of the directory creation tests
we have so far maintain grm=0 after each unmount-mount cycle. Which is
valuable.
This has, in theory, global-removes (grm) being written out as a part of
of directory creation, but they aren't used in any form and so may not
be being written correctly.
But it did require quite a bit of problem solving to get to this point
(the interactions between mtree splitsand grms is really annoying), so
it's worth a commit.
lfsr_stat is really a directory operation underneath, so it's good to
add to our testing while we are building up the dir tests.
It's interesting to note lfsr_stat and lfsr_dir_read are less
deduplicatable than their previous versions, since lfsr_stat can get
most of it's info from lfsr_mtree_pathlookup. Though there will probably
need to be some code sharing when we get to files with sizes.
This makes it now possible to create directories in the new system.
The new system now uses a single global "mtree" to store all metadata
entries in the filesystem. In this system, a directory is simply a range
of metadata entries. This has a number of benefits, but does come with
its own problems:
1. We need to indicate which directory each file belongs to. To do this
the file's name entry has been changed to a tuple of leb128-encoded
directory-id + actual file name:
01 66 69 6c 65 2e 74 78 74 .file.txt
^ '----------+----------'
'------------|------------ leb128 directory-id
'------------ ascii/utf8 name
If we include the directory-id as part of filename comparison, files
should naturally be next to other files in the same directory.
2. We need a way allocate directory-ids for new directories. This turns
out to be a bit more tricky than I expected.
We can't use any mid/bid/rid inherent to the mtree, because these
change on any file creation/deletion. And since we commit the did
into the tree, that's not acceptable.
Initially I though you could just find the largest did and increment,
but this gives you no way to reclaim deleted dids. And sure, deleted
dids have no storage consumption, but eventually you will overflow
the did integer. Since this can suddenly happen in a filesystem
that's been in a steady-state for years, that's pretty unnacceptable.
One solution is to do a simple linear search over the mtree for an
unused did. But with a runtime of O(n^2 log(n)), this raises
performance concerns.
Sidenote: It's interesting to note that the Linux kernel's allocation
of process-ids, a very similar problem, is surprisingly complex and
relies on a radix-tree of bitmaps (struct idr). This suggests I'm not
missing an obvious solution somewhere.
The solution I settled on here is to instead treat the set of dids as
a sort of hash table:
1. Hash the full directory path into a did.
2. Perform a linear search until we have no collision.
leb128(truncate28(crc32c("dir")))
.--------'
v
9e cd c8 30 66 69 6c 65 2e 74 78 74 ...0file.txt
'----+----' '----------+----------'
'-----------------|------------ leb128 directory-id
'------------ ascii/utf8 name
Worst case, this can still exhibit the worst case O(n^2 log(n))
performance when we are close to full dids. However that seems
unlikely to happen in practice, since we don't truncate our hashes,
unlike normal hash tables. An additional 32-bit word for each file
is a small price to pay for a low-chance of collisions.
In the current implementation, I do truncate the hash to 28-bits.
Since we encode the hash with leb128, and hashes are statistically
random, this gives us better usage of the leb128 encoding. However
it does limit a 32-bit littlefs to 256 Mi directories.
Maybe this should be a configurable limit in the future.
But that highlights another benefit of this scheme. It's easy to
change in the future without disk changes.
3. We need a way to know if a directory-id is allocated, even if the
directory is empty.
For this we just introduce a new tag: LFSR_TAG_DSTART, which
is an empty file entry that indicates the directory at the given did
in the mtree is allocated.
To create/delete these atomically with the reference in our parent
directory, we can use the GRM system for atomic renames.
Note this isn't implemented yet.
This is also the first time we finally get around to testing all of the
dname lookup functions, so this did find a few bugs, mostly around
reporting the root correctly.
Took the opportunity to make some allocator tweaks:
- Renamed lfs.free -> lfs.lookahead, it's previous name did cause some
confusion.
- Renamed lfs.free.off -> lfs.lookahead.start
- Renamed lfs.free.i -> lfs.lookahead.next
- Renamed lfs.free.ack -> lfs.lookahead.acked
- Changed bitmap from using 32-bit words to using 8-bit bytes, dropping
the alignment requirement. One of the reasons for 32-bit alignment was
an attempt at future proofing for some sort of free-list.
This never landed, and if it did, it could have been provided without
breaking backwards compatiblity via an additional config option, at a
minor RAM cost.
We never used ffs/clz instructions for this bitmap, so I don't think
using 32-bit words offers much advantage. It just creates another
potential issue for users if their lookahead buffer is unaligned.
These changes should probably also be upstreamed to the current version.
They don't depend on anything rbyd specific.
Note, at some point lfs_alloc will need to be extended to mark block tags,
etc, as in-use during traversal.
- Renamed mpair -> mptr, may have >2 blocks in the future.
- Renamed branch -> bptr for consistency.
- Renamed other_block -> redund_rbyd.
- Changed comparison functions to use -1, 0, +1, even for unordered
types.
- Added lfs_cmp function for unioning comparisons with signed errors.
Now we read the revision count on-demand, trading off some extra reads
for a smaller lfsr_rbyd_t struct.
I believe this is worth it because:
1. We're created a lot of lfsr_rbyd_t structs as a part of the relatively
complicated mdir/btree commit logic in order to safely fallback on errors.
2. We don't really need the revision count for our Cow btrees, so we
only need to read the revision count on mdir fetch (which we were
already reading too many times), on mdir compact, and on rbyd fetch
as a part of checksum calculation.
This really only adds a O(1) cost when we are compacting, which is rather
small.
Current measurements:
code: 8980 -> 9036 (+0.6%)
stack: 1024 -> 1000 (-2.3%)
Though note this is currently without any mdir/btree commit code being
dragged in.
littlefs uses an invasive linked-list in open mdirs to keep any open
files/dirs (and some special mdirs) in sync during filesystem
operations. The main benefit of this is that the filesystem doesn't need
to know the number of open files at compile time.
The implementation here introduces a new type, lfsr_openedmdir_t, for
mdirs that want to participate in the opened-mdir linked-list. This
saves a couple words of memory in the cases where the mdir does not need
to participate in the opend-mdir linked-list.
Since we are creating quite a few more mdir structs in lfsr_mdir_commit now,
the size of this struct is valuable.
The implementation of lfsr_mdir_commit knew this was coming, so aside
from the new type, adding this feature was straightforward:
1. Update opened-mdirs based on in-flight attrs.
2. Update opened-mdirs rbyd state.
3. Mark any deleted opened-mdirs with the reserved mid -2.
4. Test.
mdirs behave a bit differently than btree nodes here. When an mdir's
weight drops to zero, we eagerly drop the mdir. Unfortunately this
introduce a large number of conditions into lfsr_mdir_commit. Maybe
there's some different way to structure to code to avoid this...
Also expanded mtree tests to cover more corner cases, these are
desperately for any confidence that mdir drops work.
Christopher Haster