Reworked revision count logic a bit, block_cycles -> block_recycles

The original goal here was to restore all of the revision count/
wear-leveling features that were intentionally ignored during
refactoring, but over time a few other ideas to better leverage our
revision count bits crept in, so this is sort of the amalgamation of
that...

Note! None of these changes affect reading. mdir fetch strictly needs
only to look at the revision count as a big 32-bit counter to determine
which block is the most recent.

The interesting thing about the original definition of the revision
count, a simple 32-bit counter, is that it actually only needs 2-bits to
work. Well, three states really: 1. most recent, 2. less recent, 3.
future most recent. This means the remaining bits are sort of up for
grabs to other things.

Previously, we've used the extra revision count bits as a heuristic for
wear-leveling. Here we reintroduce that, a bit more rigorously, while
also carving out space for a nonce to help with commit collisions.

Here's the new revision count breakdown:

  vvvvrrrr rrrrrrnn nnnnnnnn nnnnnnnn
  '-.''----.----''---------.--------'
    '------|---------------|---------- 4-bit relocation revision
           '---------------|---------- recycle-bits recycle counter
                           '---------- pseudorandom nonce

- 4-bit relocation revision

  We technically only need 2-bits to tell which block is the most
  recent, but I've bumped it up to 4-bits just to be safe and to make
  it a bit more readable in hex form.

- recycle-bits recycle counter

  A user configurable counter, this counter tracks how many times a
  metadata block has been erased. When it overflows we return the block
  to the allocator to participate in block-level wear-leveling again.
  This implements our copy-on-bounded-write strategy.

- pseudorandom nonce

  The remaining bits we fill with a pseudorandom nonce derived from the
  filesystem's prng. Note this prng isn't the greatest (it's just the
  xor of all mdir cksums), but it gets the job done. It should also be
  reproducible, which can be a good thing.

  Suggested by ithinuel, the addition of a nonce should help with the
  commit collision issue caused by noop erases. It doesn't completely
  solve things, since we're only using crc32c cksums not collision
  resistant cryptographic hashes, but we still have the existing
  valid/perturb bit system to fall back on.

When we allocate a new mdir, we want to zero the recycle counter. This
is where our relocation revision is useful for indicating which block is
the most recent:

  initial state: 10101010 10101010 10101010 10101010
                 '-.'
                  +1     zero           random
                   v .----'----..---------'--------.
  lfsr_rev_init: 10110000 00000011 01110010 11101111

When we increment, we increment recycle counter and xor in a new nonce:

  initial state: 10110000 00000011 01110010 11101111
                 '--------.----''---------.--------'
                         +1              xor <-- random
                          v               v
  lfsr_rev_init: 10110000 00000111 01010100 01000000

And when the recycle counter overflows, we relocate the mdir.

If we aren't wear-leveling, we just increment the relocation revision to
maximize the nonce.

---

Some other notes:

- Renamed block_cycles -> block_recycles.

  This is intended to help avoid confusing block_cycles with the actual
  physical number of erase cycles supported by the device.

  I've noticed this happening a few times, and it's unfortunately
  equivalent to disabling wear-leveling completely. This can be improved
  with better documentation, but also changing the name doesn't hurt.

- We now relocate both blocks in the mdir at the same time.

  Previously we only relocated one block in the mdir per recycle. This
  was necessary to keep our threaded linked-list in sync, but the
  threaded linked-list is now no more!

  Relocating both blocks is simpler, updates the mtree less often,
  compatible with metadata redundancy, and avoids aliasing issues that
  were a problem when relocating one block.

  Note that block_recycles is internally multiplied by 2 so each block
  sees the correct number of erase cycles.

- block_recycles is now rounded down to a power-of-2.

  This makes the counter logic easier to work with and takes up less RAM
  in lfs_t. This is a rough heuristic anyways.

- Moved the lfs->seed updates into lfsr_mountinited + lfsr_mdir_commit.

  This avoids readonly operations affecting the seed and should help
  reproducibility.

- Changed rev count in dbg scripts to render as hex, similar to cksums.

  Now that we using most of the bits in the revision count, the decimal
  version is, uh, not helpful...

Code changes:

           code          stack
  before: 33342           2640
  after:  33434 (+0.3%)   2640 (+0.0%)
This commit is contained in:
Christopher Haster
2024-05-21 12:16:41 -05:00
parent 4208aa21e2
commit 56b18dfd9a
9 changed files with 107 additions and 58 deletions
+75 -27
View File
@@ -2177,12 +2177,6 @@ static int lfsr_rbyd_fetch(lfs_t *lfs, lfsr_rbyd_t *rbyd,
break;
}
// toss our cksum into the filesystem seed for
// pseudorandom numbers, note we use another cksum here
// as a collection function because it is sufficiently
// random and convenient
lfs->seed = lfs_crc32c(lfs->seed, &cksum, sizeof(uint32_t));
// save what we've found so far
rbyd->eoff
= ((lfs_size_t)parity_ << (8*sizeof(lfs_size_t)-1))
@@ -5541,6 +5535,45 @@ static int lfsr_fs_consumegdelta(lfs_t *lfs, const lfsr_mdir_t *mdir) {
}
/// Revision count things ///
// in mdirs, our revision count is broken down into three parts:
//
// vvvvrrrr rrrrrrnn nnnnnnnn nnnnnnnn
// '-.''----.----''---------.--------'
// '------|---------------|---------- 4-bit relocation revision
// '---------------|---------- recycle-bits recycle counter
// '---------- pseudorandom nonce
static inline uint32_t lfsr_rev_init(lfs_t *lfs, uint32_t rev) {
// we really only care about the top revision bits here
rev &= ~((1 << 28)-1);
// increment revision
rev += 1 << 28;
// xor in a pseudorandom nonce
rev ^= ((1 << (28-lfs_smax32(lfs->recycle_bits, 0)))-1) & lfs->seed;
return rev;
}
static inline bool lfsr_rev_needsrelocation(lfs_t *lfs, uint32_t rev) {
if (lfs->recycle_bits == -1) {
return false;
}
// does out recycle counter overflow?
uint32_t rev_ = rev + (1 << (28-lfs_smax32(lfs->recycle_bits, 0)));
return (rev_ >> 28) != (rev >> 28);
}
static inline uint32_t lfsr_rev_inc(lfs_t *lfs, uint32_t rev) {
// increment recycle counter/revision
rev += 1 << (28-lfs_smax32(lfs->recycle_bits, 0));
// xor in a pseudorandom nonce
rev ^= ((1 << (28-lfs_smax32(lfs->recycle_bits, 0)))-1) & lfs->seed;
return rev;
}
/// Metadata pair stuff ///
@@ -5868,11 +5901,8 @@ static int lfsr_mdir_alloc__(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_smid_t mid) {
// note we allow corrupt errors here, as long as they are consistent
rev = (err != LFS_ERR_CORRUPT) ? lfs_fromle32_(&rev) : 0;
// align revision count in new mdirs to our block_cycles, this makes
// sure we don't immediately try to relocate the mdir
if (lfs->cfg->block_cycles > 0) {
rev = lfs_alignup(rev+1, lfs->cfg->block_cycles)-1;
}
// reset recycle bits in revision count and increment
rev = lfsr_rev_init(lfs, rev);
// erase, preparing for compact
err = lfsr_bd_erase(lfs, mdir->rbyd.blocks[0]);
@@ -5880,9 +5910,8 @@ static int lfsr_mdir_alloc__(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_smid_t mid) {
return err;
}
// increment our revision count and write it to our rbyd
// TODO rev things
err = lfsr_rbyd_appendrev(lfs, &mdir->rbyd, rev + 1);
// write our revision count
err = lfsr_rbyd_appendrev(lfs, &mdir->rbyd, rev);
if (err) {
return err;
}
@@ -5906,10 +5935,7 @@ static int lfsr_mdir_swap__(lfs_t *lfs, lfsr_mdir_t *mdir_,
rev = (err != LFS_ERR_CORRUPT) ? lfs_fromle32_(&rev) : 0;
// decide if we need to relocate
if (!force
&& lfs->cfg->block_cycles > 0
// TODO rev things
&& (rev + 1) % lfs->cfg->block_cycles == 0) {
if (!force && lfsr_rev_needsrelocation(lfs, rev)) {
// alloc a new mdir
return lfsr_mdir_alloc__(lfs, mdir_, mdir->mid);
}
@@ -5929,8 +5955,7 @@ static int lfsr_mdir_swap__(lfs_t *lfs, lfsr_mdir_t *mdir_,
}
// increment our revision count and write it to our rbyd
// TODO rev things
err = lfsr_rbyd_appendrev(lfs, &mdir_->rbyd, rev + 1);
err = lfsr_rbyd_appendrev(lfs, &mdir_->rbyd, lfsr_rev_inc(lfs, rev));
if (err) {
return err;
}
@@ -7037,6 +7062,9 @@ static int lfsr_mdir_commit(lfs_t *lfs, lfsr_mdir_t *mdir,
// success? update in-device state, we must not error at this point
// toss our cksum into the filesystem seed for pseudorandom numbers
lfs->seed ^= mdir_.rbyd.cksum;
// update any gstate changes
lfsr_fs_commitgdelta(lfs);
@@ -8346,6 +8374,10 @@ static int lfsr_mountinited(lfs_t *lfs) {
}
}
// toss our cksum into the filesystem seed for pseudorandom
// numbers
lfs->seed ^= tinfo.u.mdir.rbyd.cksum;
// collect any gdeltas from this mdir
err = lfsr_fs_consumegdelta(lfs, &tinfo.u.mdir);
if (err) {
@@ -15248,14 +15280,17 @@ static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
// // check that the block size is large enough to fit ctz pointers
// LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4))
// <= lfs->cfg->block_size);
//
// // block_cycles = 0 is no longer supported.
// //
// // block_cycles is the number of erase cycles before littlefs evicts
// // metadata logs as a part of wear leveling. Suggested values are in the
// // range of 100-1000, or set block_cycles to -1 to disable block-level
// // wear-leveling.
// LFS_ASSERT(lfs->cfg->block_cycles != 0);
// block_cycles = 0 is no longer supported.
//
// block_cycles is the number of erase cycles before littlefs evicts
// metadata logs as a part of wear leveling. Suggested values are in the
// range of 100-1000, or set block_cycles to -1 to disable block-level
// wear-leveling.
LFS_ASSERT(lfs->cfg->block_cycles != 0);
// block_recycles should not be zero, use -1 to disable
LFS_ASSERT(lfs->cfg->block_recycles != 0);
// inline_size must be <= block_size/4
LFS_ASSERT(lfs->cfg->inline_size <= lfs->cfg->block_size/4);
@@ -15341,6 +15376,19 @@ static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
// TODO do we need to recalculate these after mount?
// find the number of bits to use for recycle counters
//
// Multiply by 2, since we alternate which metadata block we erase each
// compaction, and limit to 28-bits so we always have some bits to
// determine the most recent revision.
if (lfs->cfg->block_recycles != -1) {
lfs->recycle_bits = lfs_min(
lfs_nlog2(2*lfs->cfg->block_recycles+1)-1,
28);
} else {
lfs->recycle_bits = -1;
}
// calculate the upper-bound cost of a single rbyd attr after compaction
//
// Note that with rebalancing during compaction, we know the number
+6 -5
View File
@@ -206,13 +206,13 @@ struct lfs_config {
// Number of erasable blocks on the device.
lfs_size_t block_count;
// Number of erase cycles before littlefs evicts metadata logs and moves
// the metadata to another block. Suggested values are in the
// range 100-1000, with large values having better performance at the cost
// of less consistent wear distribution.
// Number of erase cycles before metadata blocks are relocated for
// wear-leveling. Suggested values are in the range 16-1024. Larger values
// relocate less frequently, improving average performance, at the cost
// of worse wear distribution. Note this is rounded down to a power-of-2.
//
// Set to -1 to disable block-level wear-leveling.
int32_t block_cycles;
int32_t block_recycles;
// Size of the read cache in bytes. Larger buffers can improve
// performance by storing more data and reducing the number of disk
@@ -581,6 +581,7 @@ typedef struct lfs {
// purpose flags field? this has been useful for lfsr_file_t
bool hasorphans;
int8_t recycle_bits;
uint8_t attr_estimate;
uint8_t mleaf_bits;
+2 -2
View File
@@ -115,6 +115,7 @@ void bench_permutation(size_t i, uint32_t *buffer, size_t size);
BENCH_DEFINE(BLOCK_SIZE, 4096 ) \
BENCH_DEFINE(BLOCK_COUNT, DISK_SIZE/BLOCK_SIZE ) \
BENCH_DEFINE(DISK_SIZE, 1024*1024 ) \
BENCH_DEFINE(BLOCK_RECYCLES, -1 ) \
BENCH_DEFINE(RCACHE_SIZE, LFS_MAX(16, READ_SIZE) ) \
BENCH_DEFINE(PCACHE_SIZE, LFS_MAX(16, PROG_SIZE) ) \
BENCH_DEFINE(FBUFFER_SIZE, 16 ) \
@@ -123,7 +124,6 @@ void bench_permutation(size_t i, uint32_t *buffer, size_t size);
BENCH_DEFINE(SHRUB_SIZE, INLINE_SIZE ) \
BENCH_DEFINE(FRAGMENT_SIZE, BLOCK_SIZE/8 ) \
BENCH_DEFINE(CRYSTAL_THRESH, BLOCK_SIZE/8 ) \
BENCH_DEFINE(BLOCK_CYCLES, -1 ) \
BENCH_DEFINE(ERASE_VALUE, 0xff ) \
BENCH_DEFINE(ERASE_CYCLES, 0 ) \
BENCH_DEFINE(BADBLOCK_BEHAVIOR, LFS_EMUBD_BADBLOCK_PROGERROR ) \
@@ -142,7 +142,7 @@ void bench_permutation(size_t i, uint32_t *buffer, size_t size);
.prog_size = PROG_SIZE, \
.block_size = BLOCK_SIZE, \
.block_count = BLOCK_COUNT, \
.block_cycles = BLOCK_CYCLES, \
.block_recycles = BLOCK_RECYCLES, \
.rcache_size = RCACHE_SIZE, \
.pcache_size = PCACHE_SIZE, \
.fbuffer_size = FBUFFER_SIZE, \
+2 -2
View File
@@ -99,6 +99,7 @@ void test_permutation(size_t i, uint32_t *buffer, size_t size);
TEST_DEFINE(BLOCK_SIZE, 4096 ) \
TEST_DEFINE(BLOCK_COUNT, DISK_SIZE/BLOCK_SIZE ) \
TEST_DEFINE(DISK_SIZE, 1024*1024 ) \
TEST_DEFINE(BLOCK_RECYCLES, -1 ) \
TEST_DEFINE(RCACHE_SIZE, LFS_MAX(16, READ_SIZE) ) \
TEST_DEFINE(PCACHE_SIZE, LFS_MAX(16, PROG_SIZE) ) \
TEST_DEFINE(FBUFFER_SIZE, 16 ) \
@@ -107,7 +108,6 @@ void test_permutation(size_t i, uint32_t *buffer, size_t size);
TEST_DEFINE(SHRUB_SIZE, INLINE_SIZE ) \
TEST_DEFINE(FRAGMENT_SIZE, BLOCK_SIZE/8 ) \
TEST_DEFINE(CRYSTAL_THRESH, BLOCK_SIZE/8 ) \
TEST_DEFINE(BLOCK_CYCLES, -1 ) \
TEST_DEFINE(ERASE_VALUE, 0xff ) \
TEST_DEFINE(ERASE_CYCLES, 0 ) \
TEST_DEFINE(BADBLOCK_BEHAVIOR, LFS_EMUBD_BADBLOCK_PROGERROR ) \
@@ -126,7 +126,7 @@ void test_permutation(size_t i, uint32_t *buffer, size_t size);
.prog_size = PROG_SIZE, \
.block_size = BLOCK_SIZE, \
.block_count = BLOCK_COUNT, \
.block_cycles = BLOCK_CYCLES, \
.block_recycles = BLOCK_RECYCLES, \
.rcache_size = RCACHE_SIZE, \
.pcache_size = PCACHE_SIZE, \
.fbuffer_size = FBUFFER_SIZE, \
+1 -1
View File
@@ -610,7 +610,7 @@ def main(disk, roots=None, *,
# fetch the root
btree = Rbyd.fetch(f, block_size, roots, trunk)
print('btree %s, rev %d, weight %d, cksum %08x' % (
print('btree %s, rev %08x, weight %d, cksum %08x' % (
btree.addr(), btree.rev, btree.weight, btree.cksum))
# look up a bid, while keeping track of the search path
+1 -1
View File
@@ -1884,7 +1884,7 @@ def main(disk, mroots=None, *,
#### actual debugging begins here
# print some information about the filesystem
print('littlefs v%s.%s %dx%d %s, rev %d, weight %d.%d' % (
print('littlefs v%s.%s %dx%d %s, rev %08x, weight %d.%d' % (
config.version[0] if config.version[0] is not None else '?',
config.version[1] if config.version[1] is not None else '?',
(config.geometry[0] or 0), (config.geometry[1] or 0),
+1 -1
View File
@@ -1500,7 +1500,7 @@ def main(disk, mroots=None, *,
#### actual debugging begins here
# print some information about the mtree
print('mtree %s, rev %d, weight %d.%d, cksum %08x' % (
print('mtree %s, rev %08x, weight %d.%d, cksum %08x' % (
mroot.addr(),
mroot.rev,
bweight//mleaf_weight, 1*mleaf_weight,
+1 -1
View File
@@ -1030,7 +1030,7 @@ def main(disk, blocks=None, *,
weights[i],
cksums[i])
print('rbyd %s, rev %d, size %d, weight %d, cksum %08x' % (
print('rbyd %s, rev %08x, size %d, weight %d, cksum %08x' % (
'0x%x.%x' % (block, trunk_)
if len(blocks) == 1
else '0x{%x,%s}.%x' % (
+18 -18
View File
@@ -1136,7 +1136,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1221,7 +1221,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1318,7 +1318,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1415,7 +1415,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1469,7 +1469,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
# force our block to compact by setting prog_size=block_size, we don't have
# an easy way to force the intermediary mroots to compact otherwise
defines.PROG_SIZE = 'BLOCK_SIZE'
@@ -1540,7 +1540,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1603,7 +1603,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1705,7 +1705,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1824,7 +1824,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1914,7 +1914,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -1990,7 +1990,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -2076,7 +2076,7 @@ code = '''
[cases.test_mtree_relocate_fuzz]
defines.N = [5, 10, 20, 40]
defines.FORCE_COMPACTION = [false, true]
defines.BLOCK_CYCLES = [5, 2, 1]
defines.BLOCK_RECYCLES = [5, 2, 1]
defines.SEED = 'range(500)'
in = 'lfs.c'
code = '''
@@ -2555,7 +2555,7 @@ code = '''
[cases.test_mtree_opened_extend]
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -2613,7 +2613,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -2702,7 +2702,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -3528,7 +3528,7 @@ code = '''
[cases.test_mtree_traversal_extend]
defines.VALIDATE = [false, true]
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -4003,7 +4003,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
in = 'lfs.c'
code = '''
lfs_t lfs;
@@ -4057,7 +4057,7 @@ code = '''
# this should be set so only one entry can fit in a metadata block
defines.SIZE = 'BLOCK_SIZE / 4'
# make it so blocks relocate every two compacts
defines.BLOCK_CYCLES = 2
defines.BLOCK_RECYCLES = 1
# force our block to compact by setting prog_size=block_size, we don't have
# any way to indirectly force the intermediary mroots to compact otherwise
defines.PROG_SIZE = 'BLOCK_SIZE'