The grm bugs were mostly issues with:
1. Not maintaining the on-disk grm state in RAM (lfs->grm) correctly,
this needs to be updated correctly after every commit or littlefs
gets a confused.
2. lfsr_fs_fixgrm got a bit confused when it was missed when changing
the no-rm encoding from 0 to -2. Added some inline functions to help
avoid this in the future.
3. Leaking information due to mixing fixed sized and variable sized
encodings of the grm delta in places. This is a bit tricky to write
an assert for as we don't parse the full grm when we see a no-rm grm.
This implementation is in theory correct, but of course, being untested,
who knows?
Though this does come with remounting added to all of the directory
tests. This effectively tests that all of the directory creation tests
we have so far maintain grm=0 after each unmount-mount cycle. Which is
valuable.
This bug was just overlooked in testing the mtree, fortunately dir
fuzzing found it. Though since this depends on neighboring mdirs, it
probably would have been found quicker with smaller block sizes. At the
moment I am only testing on NOR-liked geometry (4KiB blocks).
The fix is easy, we can use the difference in the mtree size to
determine if a split or drop happened in mdir commit, since at most one
of these can happen on any mdir commit.
Also added an explicit test for mid updates when splitting and dropping.
lfsr_stat is really a directory operation underneath, so it's good to
add to our testing while we are building up the dir tests.
It's interesting to note lfsr_stat and lfsr_dir_read are less
deduplicatable than their previous versions, since lfsr_stat can get
most of it's info from lfsr_mtree_pathlookup. Though there will probably
need to be some code sharing when we get to files with sizes.
- Checksum collisions
- Collisions with root did
- Collisions needing wraparound
- Possible leb128 encoding issues
Sure enough the last one caught an off-by-one error in our calculation
of the leb128 encoded size. I sort of expected a bug there, since it's
rather nuanced math, so it's good to have test coverage now.
The main issues:
- The addition of the root's dstart entry during lfsr_format throws off
our mtree tests. It's a bit of a hack, but for now I am just manually
deleting the root's dstart entry at the beginning of each tests.
It might be possible to make the mtree tests work around the root's
dstart, but it seems to cause problems for when exactly the mtree
splits.
- btree dnamelookup and mdir dnamelookup need different things from
the rbyd dnamelookup when the dname is not found. The btree lookup
needs the largest branch smaller than the dname, since this is the
"bucket" containing our dname, while the mdir dnamelookup needs
the id that _follows_ the id smaller than the dname, since insertion
causes all ids >= the inserting id to shift up.
The solution here is to make rbyd dnamelookup behave as expected by
btree dnamelookup. btree needs more info about the branch (weight
mostly), so this avoids more issues. mdir dnamelookup adjusts the
id as needed, which costs a bit of code, but makes things work.
Fortunately, mdir dnamelookup can assume the weight is 1, which
simplifies things a bit.
This makes it now possible to create directories in the new system.
The new system now uses a single global "mtree" to store all metadata
entries in the filesystem. In this system, a directory is simply a range
of metadata entries. This has a number of benefits, but does come with
its own problems:
1. We need to indicate which directory each file belongs to. To do this
the file's name entry has been changed to a tuple of leb128-encoded
directory-id + actual file name:
01 66 69 6c 65 2e 74 78 74 .file.txt
^ '----------+----------'
'------------|------------ leb128 directory-id
'------------ ascii/utf8 name
If we include the directory-id as part of filename comparison, files
should naturally be next to other files in the same directory.
2. We need a way allocate directory-ids for new directories. This turns
out to be a bit more tricky than I expected.
We can't use any mid/bid/rid inherent to the mtree, because these
change on any file creation/deletion. And since we commit the did
into the tree, that's not acceptable.
Initially I though you could just find the largest did and increment,
but this gives you no way to reclaim deleted dids. And sure, deleted
dids have no storage consumption, but eventually you will overflow
the did integer. Since this can suddenly happen in a filesystem
that's been in a steady-state for years, that's pretty unnacceptable.
One solution is to do a simple linear search over the mtree for an
unused did. But with a runtime of O(n^2 log(n)), this raises
performance concerns.
Sidenote: It's interesting to note that the Linux kernel's allocation
of process-ids, a very similar problem, is surprisingly complex and
relies on a radix-tree of bitmaps (struct idr). This suggests I'm not
missing an obvious solution somewhere.
The solution I settled on here is to instead treat the set of dids as
a sort of hash table:
1. Hash the full directory path into a did.
2. Perform a linear search until we have no collision.
leb128(truncate28(crc32c("dir")))
.--------'
v
9e cd c8 30 66 69 6c 65 2e 74 78 74 ...0file.txt
'----+----' '----------+----------'
'-----------------|------------ leb128 directory-id
'------------ ascii/utf8 name
Worst case, this can still exhibit the worst case O(n^2 log(n))
performance when we are close to full dids. However that seems
unlikely to happen in practice, since we don't truncate our hashes,
unlike normal hash tables. An additional 32-bit word for each file
is a small price to pay for a low-chance of collisions.
In the current implementation, I do truncate the hash to 28-bits.
Since we encode the hash with leb128, and hashes are statistically
random, this gives us better usage of the leb128 encoding. However
it does limit a 32-bit littlefs to 256 Mi directories.
Maybe this should be a configurable limit in the future.
But that highlights another benefit of this scheme. It's easy to
change in the future without disk changes.
3. We need a way to know if a directory-id is allocated, even if the
directory is empty.
For this we just introduce a new tag: LFSR_TAG_DSTART, which
is an empty file entry that indicates the directory at the given did
in the mtree is allocated.
To create/delete these atomically with the reference in our parent
directory, we can use the GRM system for atomic renames.
Note this isn't implemented yet.
This is also the first time we finally get around to testing all of the
dname lookup functions, so this did find a few bugs, mostly around
reporting the root correctly.
Christopher Haster