Adopted a tighter, block-size dependendent attr-estimate

The concern right now is small-block filesystems, anything in the 512B
to <4KiB range. With such small blocks, and rbyd's relatively high
per-attr overhead, there's a real risk that littlefs may just not be
able to function without quickly running to metadata limits.

I realize these are pretty rare geometries for flash, but they are still
common for anything that 1. pretends to be a spinny disks, SD cards,
FTLs, eMMCs, etc, and 2. mapping into RAM, which is surprisingly common.

It is possible to require this sort of geometry to pretend to be a
larger logical block-size, but since this is a regression from the
previous version of littlefs, it would be nice to avoid this if
possible.

Anyways, what actually is this commit. Consider our tag encoding:

  .---+---+---+- -+- -+- -+- -+---+- -+- -+- -.  tag:    2 bytes
  |  tag  | weight            | size          |  weight: <=5 bytes
  '---+---+---+- -+- -+- -+- -+---+- -+- -+- -'  size:   <=4 bytes
                                                 total:  <=11 bytes

With our current 32-bit (really 31-bit) version of littlefs, the worst
case tag encoding is 11 bytes.

This doesn't sound that bad, but with our current compaction algorithm we
need ~2.5 tags for each attr:

        5t       5*11
  a_1 = -- + 2 = ---- = 30 bytes
         2         2

Are there any additional assumptions we can make to push our attr
estimate lower?

- tag - Ignoring a complete redesign of our tag encoding (which has
  already been heavily iterated over), this just needs 2 bytes, which is
  not that bad.

- weight - This is the real painful one because, for the most part,
  weight=0. But weight _can_ store a full size, in the case it is the
  root of a file's btree. So this is pretty much stuck at an annoying
  5 bytes.

  I suppose this could be tied to our size-limit. I hadn't thought about
  that until writing this commit message. Maybe that can be a future
  improvement, though it won't really have a big effect on most systems.

- size/jump - Now this field is interesting. When expressing both the
  size of tag payloads, and the relative jump offset for alt-pointers,
  this field should never exceed a single block.

  We've already pushed this down to 4 bytes at compile time, by assuming
  at most 28-bit block-sizes, but if we know the block-size, we could in
  theory push this even lower.

  This is extra enticing, because the block-sizes where the size/jump
  field can be shrunk, are _also_ the block-sizes where the metadata
  density is so critical!

So that's what this commit does. For the purpose of compaction estimates
(not stack allocations!) we calculate attr estimate based on our
runtime-determined block_size.

Here are some cutoff points for our new attr estimate:

  block-size      tag-estimate  attr-estimate
        512B  =>       9 bytes       25 bytes
       16KiB  =>      10 bytes       27 bytes
        2MiB  =>      11 bytes       30 bytes
      256MiB  =>      12 bytes       32 bytes

There is a question of when to actually do this calculation. We always
know our block-size, so we could recalculate the attr-estimate every
time we need to estimate a compaction. But for now I'm just
precalculating the attr estimate in lfs_init and storing in the lfs_t
struct. It's only a byte after all.

If I did my math correctly, we won't exceed a byte until we have a
block-size of 2^1750, at which point we may have other problems.

Code changes:

           code          stack          lfs_t
  before: 34068           2880            216
  after:  34104 (-0.1%)   2880 (+0.0%)    220 (+1.9%)

The jump in lfs_t cost is probably just from a word alignment boundary.

In the future, if we have compile-time block-sizes, the entire
attr-estimate could even be compile-time.
This commit is contained in:
Christopher Haster
2024-02-26 01:57:45 -06:00
parent d8d6052d90
commit 9fcf8e12d8
2 changed files with 42 additions and 33 deletions
+2
View File
@@ -598,7 +598,9 @@ typedef struct lfs {
// purpose flags field? this has been useful for lfsr_file_t
bool hasorphans;
uint8_t attr_estimate;
uint8_t mleaf_bits;
lfsr_mdir_t mroot;
lfsr_mtree_t mtree;