Github/littlefs
1
Fork 0
Code Actions Activity

gbmap: Added mkgbmap and rmgbmap for enabling/disabling the gbmap

Browse Source 
These two functions allow changing whether or not the gbmap is in use
after format:

  // Enable the global on-disk block-map
  //
  // Returns a negative error code on failure. Does nothing if a gbmap
  // already exists.
  int lfs3_fs_mkgbmap(lfs3_t *lfs3);

  // Disable the global on-disk block-map
  //
  // Returns a negative error code on failure. Does nothing if no gbmap
  // is found.
  int lfs3_fs_rmgbmap(lfs3_t *lfs3);

rmgbmap was easy enough, but implementing mkgbmap turned out to be
surprisingly tricky due to how gstate permeates the system:

- Even if we zero gstate when we removing the gbmap, mounting the
  image on a driver that doesn't understand the gbmap results in garbage
  gstate over time as mdir compacts drop unknown gdeltas.

  I think this sort of implicit gdelta cleanup is a good thing, but the
  possibility of garbage gstate is a bit annoying.

  Example A: the dbg scripts are currently printing a bunch of warnings
  for corrupt gstate that can be safely ignored.

  To support recovering from garbage gstate in mkgbmap, I changed
  lfs3_fs_commitgdelta to _always_ track p state even when disabled. We
  already needed to do this in lfs3_fs_flush/consumegdelta anyways,
  since we don't know if the gbmap is used until parsing wcompat flags.

- The commit that enables the gbmap is tricky. We need the gbmap enabled
  to calculate the new gdelta, but we also need it disabled so we don't
  traverse the existing gbmap_p (which may be garbage).

  As a workaround I added gbmap.b_p, which is in theory redundant with
  gbmap_p, but (1) avoids needing to decode gbmap_p during traversals,
  and (2) allows the two to temporarily fall out-of-sync in mkgbmap.

  This means we potentially have 5 (!) snaphots flying around when
  rebuilding the gbmap, which is starting to get a bit silly. But this
  was also motivated by gbmap_p decoding adding roughly the same amount
  of RAM to lfs3_mtree_traverse_, so the total RAM usage should in
  theory be roughly the same.

  There might be a better solution, but this at least gets mkgbmap
  working. The gbmap builds are not our most RAM senstive configurations
  anyways.

---

Also added a couple more tests in test_gbmap to test these:

- test_gbmap_files
- test_gbmap_rmgbmap
- test_gbmap_mkgbmap
- test_gbmap_rmmkgbmap
- test_gbmap_mkrmgbmap

And an explicit wraparound test to test_alloc. This was loosely implied
by the nospc tests, but it's probably better to have an explicit test.
The only downside is this implementation is limited to files:

- test_alloc_wraparound_files

---

Note we are currently dealing with three different configurations:
no-gbmap (the default), yes-gbmap (LFS3_YES_GBMAP), and maybe-gbmap
(LFS3_GBMAP + LFS3_F_GBMAP at runtime).

It only makes sense to include these in maybe-gbmap mode, so this is the
only mode with a notable code increase. However these functions are
relatively cheap. The stack/ctx changes also affect yes-gbmap, but
should mostly cancel out, see above:

                       code          stack          ctx
  no-gbmap before:    37168           2352          684
  no-gbmap after:     37168 (+0.0%)   2352 (+0.0%)  684 (+0.0%)

                       code          stack          ctx
  maybe-gbmap before: 39292           2456          800
  maybe-gbmap after:  39688 (+1.0%)   2392 (-2.6%)  852 (+6.5%)

                       code          stack          ctx
  yes-gbmap before:   39116           2456          800
  yes-gbmap after:    39156 (+0.1%)   2392 (-2.6%)  852 (+6.5%)
This commit is contained in:
Christopher Haster
2025-10-11 13:34:30 -05:00
parent 9e45249b29
commit b5a94f3397
4 changed files with 1027 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lfs3.c
+127 -24
View File
@@ -7786,10 +7786,18 @@ static void lfs3_fs_commitgdelta(lfs3_t *lfs3) {
// keep track of the on-disk grm
lfs3_data_fromgrm(&lfs3->grm, lfs3->grm_p);
// keep track of the on-disk gbmap
#ifdef LFS3_GBMAP
// keep track of the on-disk gbmap
if (lfs3_f_isgbmap(lfs3->flags)) {
// keep track of both the committed gstate and btree for
// traversals
lfs3->gbmap.b_p = lfs3->gbmap.b;
lfs3_data_fromgbmap(&lfs3->gbmap, lfs3->gbmap_p);
// if disabled, we still want to keep track of the on-disk gstate
// in case the user wants to re-enable the gbmap
} else {
lfs3_memxor(lfs3->gbmap_p, lfs3->gbmap_d, LFS3_GBMAP_DSIZE);
}
#endif
}
@@ -10387,25 +10395,12 @@ static lfs3_stag_t lfs3_mtree_traverse_(lfs3_t *lfs3, lfs3_trv_t *trv,
== LFS3_TSTATE_GBMAP,
false)) {
#ifdef LFS3_GBMAP
// decode the on-disk gbmap
//
// TODO this adds 64 bytes of mostly unused stack
// to the stack hot-path, can we avoid this somehow?
// do we care in gbmap mode?
//
lfs3_gbmap_t gbmap_p;
err = lfs3_data_readgbmap(lfs3,
&LFS3_DATA_BUF(lfs3->gbmap_p,
LFS3_GBMAP_DSIZE),
&gbmap_p);
if (err) {
LFS3_UNREACHABLE();
}
// if on-disk gbmap does not match the active gbmap,
// transition to traversing the on-disk gbmap
if (lfs3_btree_cmp(&gbmap_p.b, &lfs3->gbmap.b) != 0) {
trv->b.shrub = gbmap_p.b;
if (lfs3_btree_cmp(
&lfs3->gbmap.b_p,
&lfs3->gbmap.b) != 0) {
trv->b.shrub = lfs3->gbmap.b_p;
trv->bid = -2;
lfs3_t_settstate(&trv->b.h.flags,
LFS3_TSTATE_GBMAP_P);
@@ -10723,6 +10718,15 @@ eot:;
/// Optional on-disk block map ///
#if !defined(LFS3_RDONLY) && !defined(LFS3_2BONLY) && defined(LFS3_GBMAP)
static void lfs3_gbmap_init(lfs3_gbmap_t *gbmap) {
gbmap->window = 0;
gbmap->known = 0;
lfs3_btree_init(&gbmap->b);
lfs3_btree_init(&gbmap->b_p);
}
#endif
#if !defined(LFS3_RDONLY) && !defined(LFS3_2BONLY) && defined(LFS3_GBMAP)
static lfs3_data_t lfs3_data_fromgbmap(const lfs3_gbmap_t *gbmap,
uint8_t buffer[static LFS3_GBMAP_DSIZE]) {
@@ -10775,6 +10779,8 @@ static int lfs3_data_readgbmap(lfs3_t *lfs3, lfs3_data_t *data,
// make sure to zero btree leaf
lfs3_btree_discardleaf(&gbmap->b);
// and keep track of the committed gbmap for traversals
gbmap->b_p = gbmap->b;
return 0;
}
#endif
@@ -11310,8 +11316,6 @@ static lfs3_sblock_t lfs3_alloc(lfs3_t *lfs3, uint32_t flags) {
#if !defined(LFS3_RDONLY) && defined(LFS3_GBMAP)
static int lfs3_alloc_rebuildgbmap(lfs3_t *lfs3) {
// we should ckpoint before calling this
LFS3_ASSERT(lfs3->lookahead.ckpoint == lfs3->block_count);
LFS3_INFO("Rebuilding gbmap "
"(gbmap %"PRId32"/%"PRId32")",
lfs3->lookahead.gbmapped,
@@ -15544,6 +15548,7 @@ static int lfs3_init(lfs3_t *lfs3, uint32_t flags,
// TODO are these zeros accomplished by flushgdelta in mountinited?
// should the flushgdelta be dropped?
// TODO should we just call flushgdelta here?
// zero gstate
lfs3->gcksum = 0;
@@ -15560,11 +15565,10 @@ static int lfs3_init(lfs3_t *lfs3, uint32_t flags,
#endif
// setup other global gbmap state
// TODO is this actually needed?
#ifdef LFS3_GBMAP
lfs3_btree_init(&lfs3->gbmap.b);
lfs3->gbmap.window = 0;
lfs3->gbmap.known = 0;
lfs3_gbmap_init(&lfs3->gbmap);
// TODO should this be in the gbmap struct?
lfs3->lookahead.gbmapped = 0;
lfs3_memset(lfs3->gbmap_p, 0, LFS3_GBMAP_DSIZE);
lfs3_memset(lfs3->gbmap_d, 0, LFS3_GBMAP_DSIZE);
#endif
@@ -17112,6 +17116,105 @@ failed:;
}
#endif
// enable the global on-disk block-map
#if !defined(LFs3_RDONLY) && defined(LFS3_GBMAP) && !defined(LFS3_YES_GBMAP)
int lfs3_fs_mkgbmap(lfs3_t *lfs3) {
// do nothing if we already have a gbmap
if (lfs3_f_isgbmap(lfs3->flags)) {
return 0;
}
// prepare our filesystem for writing
int err = lfs3_fs_mkconsistent(lfs3);
if (err) {
return err;
}
// checkpoint the allocator
// TODO, should lfs3_fs_mkconsistent also checkpoint the allocator?
err = lfs3_alloc_ckpoint(lfs3);
if (err) {
return err;
}
// create an empty gbmap
lfs3_gbmap_init(&lfs3->gbmap);
// TODO should this be in the gbmap struct?
lfs3->lookahead.gbmapped = 0;
// start with everything free, rebuilding the gbmap will populate it
err = lfs3_gbmap_commit(lfs3, &lfs3->gbmap.b, 0, LFS3_RATTRS(
LFS3_RATTR(LFS3_TAG_BMFREE, +lfs3->block_count)));
if (err) {
goto failed;
}
// go ahead and mark gbmap as in-use internally
lfs3->flags |= LFS3_F_GBMAP;
// checkpoint the allocator, this should trigger a rebuild
err = lfs3_alloc_ckpoint(lfs3);
if (err) {
goto failed;
}
// mark the gbmap as in-use on-disk while atomically committing the
// gbmap into gstate
lfs3_wcompat_t wcompat_ = lfs3_wcompat(lfs3);
wcompat_ |= LFS3_WCOMPAT_GBMAP;
err = lfs3_mdir_commit(lfs3, &lfs3->mroot, LFS3_RATTRS(
LFS3_RATTR_LE32(LFS3_TAG_WCOMPAT, 0, wcompat_)));
if (err) {
goto failed;
}
return 0;
failed:;
// if we failed clear the gbmap bit and reset the gbmap to be safe
lfs3->flags &= ~LFS3_F_GBMAP;
lfs3_gbmap_init(&lfs3->gbmap);
// TODO should this be in the gbmap struct?
lfs3->lookahead.gbmapped = 0;
return err;
}
#endif
// disable the global on-disk block-map
#if !defined(LFs3_RDONLY) && defined(LFS3_GBMAP) && !defined(LFS3_YES_GBMAP)
int lfs3_fs_rmgbmap(lfs3_t *lfs3) {
// do nothing if we already don't have a gbmap
if (!lfs3_f_isgbmap(lfs3->flags)) {
return 0;
}
// prepare our filesystem for writing
int err = lfs3_fs_mkconsistent(lfs3);
if (err) {
return err;
}
// removing the gbmap is relatively easy, we just need to mark the
// gbmap as not in use
//
// this leaves garbage gdeltas around, but these should be cleaned
// up implicitly as mdirs are compacted
lfs3_wcompat_t wcompat_ = lfs3_wcompat(lfs3);
wcompat_ &= ~LFS3_WCOMPAT_GBMAP;
err = lfs3_mdir_commit(lfs3, &lfs3->mroot, LFS3_RATTRS(
LFS3_RATTR_LE32(LFS3_TAG_WCOMPAT, 0, wcompat_)));
if (err) {
return err;
}
// on success mark gbmap as not-in-use internally
lfs3->flags &= ~LFS3_F_GBMAP;
return 0;
}
#endif
/// High-level filesystem traversal ///