Github/littlefs
1
Fork 0
Code Actions Activity

Rerouted all btree mutation through attr-list parser

Browse Source 
The idea here: Instead of having unique functionality for each
individual btree operation (push/set/pop/split), we treat btrees sort of
like rbyds, with a single commit entry point that operates on attr-lists.

This adds code cost, due to needing to parse the attr-list for properties
that can affect inlined btrees (tag changes mostly), but, in theory, comes
with some advantages:

1. A single btree commit entry point with all of the inlined/uninlining
   logic should offer better chances for code deduplication, vs
   spreading this logic out in each btree operation.

2. Higher-levels should know what the current weight of the branch is,
   so we may be able to avoid the implicit math needed to calculate
   deltas.

3. Higher-levels have more knowledge about the state of the btree in
   general, so there may be other shortcuts. The mtree, for example,
   only operates on weight=1 entries, which greatly simplifies a lot of
   the related math.

Note that btrees still have strict limits in what's possible in an
attr-list. Btree operations can't cross leaf-rbyd boundaries for
example.

---

A notable omission in this change is the loss of reinlining btrees.

This wase dropped for a couple reasons. It may be worth adding back at a
later time, maybe after we actually have files implemented, but for now
does not seem worth it:

1. Reinlining adds code cost. Reinlining is more complex than you might
   expect because we only reinline on compaction. And because we compact
   before playing out our attr-list, we need to know if a commit makes
   the btree inlinable before committing to the btree.

   This is still doable with our attr-lists. We already derive the
   change in tags, since we need this to know when to uninline. But it
   adds a kind of complex bailing out of btree commits.

2. The benefits of reinlining may not be that great. In most systems, a
   tree that is uninlined once is likely to be uninlined again. It's
   only if there is a bigger state change in a system that it makes
   sense to reinline.

   Though, to be fair, waiting for compaction to reinline handled this
   quite well. Only reinlining when all erased storage is used up...

3. Thanks to our roots did entry, our mtree can never reinline.

   It would be nice to change this, but this would require explicit
   handling in lfsr_mdir_commit. Future work?

4. Files are another can of worms, with more complex interactions with
   inlinability thanks to (at least on paper right now) always having
   inlined data even when uninlined.

   If reinlining is valuable for files this can change during that work.

5. Even if files never support reinlinability, truncating files (via
   either lfsr_file_truncate or LFSR_O_TRUNC) should give the file a
   blank slate, effectively reinlining the file in that case.

---

The current implementation also changes the attr-list to be mutable so
we can adjust attr-list based on the current btree node. This is a
temporary hack! We should add the appropriate functionality to our rbyd
utilities to revert this eventually.
This commit is contained in:
Christopher Haster
2023-08-13 12:40:34 -05:00
parent 3dbc986752
commit 9b2f3cd5bb
4 changed files with 840 additions and 607 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lfs.c
+572 -349
View File
File diff suppressed because it is too large Load Diff
lfs.h
+1 -1
View File
@@ -371,7 +371,7 @@ typedef struct lfsr_btree {
lfs_size_t weight; lfs_size_t weight;
lfsr_tag_t tag; lfsr_tag_t tag;
uint16_t size; uint16_t size;
uint8_t buffer[LFSR_BTREE_INLINESIZE]; uint8_t buf[LFSR_BTREE_INLINESIZE];
} i; } i;
struct { struct {
lfsr_rbyd_t rbyd; lfsr_rbyd_t rbyd;
lfs_util.h
+10
View File
@@ -163,6 +163,7 @@ static inline int32_t lfs_smin32(int32_t a, int32_t b) {
return (a < b) ? a : b; return (a < b) ? a : b;
} }
// TODO other 16-bit ops?
static inline uint16_t lfs_max16(uint16_t a, uint16_t b) { static inline uint16_t lfs_max16(uint16_t a, uint16_t b) {
return (a > b) ? a : b; return (a > b) ? a : b;
} }
@@ -171,6 +172,15 @@ static inline uint16_t lfs_min16(uint16_t a, uint16_t b) {
return (a < b) ? a : b; return (a < b) ? a : b;
} }
// Clamp is useful as the logic for min/max when clamping can become confusing
static inline uint32_t lfs_clamp32(uint32_t a, uint32_t min, uint32_t max) {
return lfs_min32(lfs_max32(a, min), max);
}
static inline int32_t lfs_sclamp32(int32_t a, int32_t min, int32_t max) {
return lfs_smin32(lfs_smax32(a, min), max);
}
// Absolute value of signed numbers // Absolute value of signed numbers
static inline int32_t lfs_abs32(int32_t a) { static inline int32_t lfs_abs32(int32_t a) {
return a < 0 ? -a : a; return a < 0 ? -a : a;
tests/test_btree.toml
+257 -257
View File
@@ -2424,263 +2424,263 @@ code = '''
free(sim); free(sim);
''' '''
# TODO
# test we reinline (go from uninlined to inlined) correctly, this is a bit ## test we reinline (go from uninlined to inlined) correctly, this is a bit
# tricky since our btrees lazily reinline ## tricky since our btrees lazily reinline
[cases.test_btree_reinline_pop_set] #[cases.test_btree_reinline_pop_set]
in = 'lfs.c' #in = 'lfs.c'
code = ''' #code = '''
lfs_t lfs; # lfs_t lfs;
lfs_init(&lfs, CFG) => 0; # lfs_init(&lfs, CFG) => 0;
// create free lookahead # // create free lookahead
memset(lfs.lookahead.buffer, 0, CFG->lookahead_size); # memset(lfs.lookahead.buffer, 0, CFG->lookahead_size);
lfs.lookahead.start = 0; # lfs.lookahead.start = 0;
lfs.lookahead.size = lfs_min(8*CFG->lookahead_size, # lfs.lookahead.size = lfs_min(8*CFG->lookahead_size,
CFG->block_count); # CFG->block_count);
lfs.lookahead.next = 0; # lfs.lookahead.next = 0;
lfs_alloc_ack(&lfs); # lfs_alloc_ack(&lfs);
#
// create an uninlined tree # // create an uninlined tree
lfsr_btree_t btree = LFSR_BTREE_NULL; # lfsr_btree_t btree = LFSR_BTREE_NULL;
lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1,
LFSR_DATA("a", 1)) => 0; # LFSR_DATA("a", 1)) => 0;
lfsr_btree_push(&lfs, &btree, 1, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, 1, LFSR_TAG_INLINED, 1,
LFSR_DATA("b", 1)) => 0; # LFSR_DATA("b", 1)) => 0;
assert(lfsr_btree_weight(&btree) == 2); # assert(lfsr_btree_weight(&btree) == 2);
assert(!lfsr_btree_isinlined(&btree)); # assert(!lfsr_btree_isinlined(&btree));
#
// pop! our btree should now be reinlinable # // pop! our btree should now be reinlinable
lfsr_btree_pop(&lfs, &btree, 0) => 0; # lfsr_btree_pop(&lfs, &btree, 0) => 0;
#
// but thanks to lazy reinlining, our btree won't reinline until # // but thanks to lazy reinlining, our btree won't reinline until
// it is compacted, so we need to add commits until it is compacted # // it is compacted, so we need to add commits until it is compacted
lfs_block_t before_block = btree.u.r.rbyd.block; # lfs_block_t before_block = btree.u.r.rbyd.block;
for (lfs_block_t i = 0;; i++) { # for (lfs_block_t i = 0;; i++) {
// a bit hacky, but this catches infinite loops # // a bit hacky, but this catches infinite loops
assert(i < BLOCK_SIZE); # assert(i < BLOCK_SIZE);
#
// commit to btree # // commit to btree
lfsr_btree_set(&lfs, &btree, 0, LFSR_TAG_INLINED, 1, # lfsr_btree_set(&lfs, &btree, 0, LFSR_TAG_INLINED, 1,
LFSR_DATA("b", 1)) => 0; # LFSR_DATA("b", 1)) => 0;
#
assert(lfsr_btree_weight(&btree) == 1); # assert(lfsr_btree_weight(&btree) == 1);
#
// try looking up tag to hopefully catch if something breaks # // try looking up tag to hopefully catch if something breaks
uint8_t buffer[4]; # uint8_t buffer[4];
lfsr_tag_t tag_; # lfsr_tag_t tag_;
lfs_size_t weight_; # lfs_size_t weight_;
#
lfsr_btree_get(&lfs, &btree, 0, # lfsr_btree_get(&lfs, &btree, 0,
&tag_, &weight_, buffer, 4) => 1; # &tag_, &weight_, buffer, 4) => 1;
assert(tag_ == LFSR_TAG_INLINED); # assert(tag_ == LFSR_TAG_INLINED);
assert(weight_ == 1); # assert(weight_ == 1);
assert(memcmp(buffer, "b", 1) == 0); # assert(memcmp(buffer, "b", 1) == 0);
#
// inlined? consider this a success # // inlined? consider this a success
if (lfsr_btree_isinlined(&btree)) { # if (lfsr_btree_isinlined(&btree)) {
break; # break;
} # }
#
// assert if a compaction occurred that wasn't inlined # // assert if a compaction occurred that wasn't inlined
assert(btree.u.r.rbyd.block == before_block); # assert(btree.u.r.rbyd.block == before_block);
} # }
#
printf("btree: w%d 0x%x.%x\n", # printf("btree: w%d 0x%x.%x\n",
btree.u.r.rbyd.weight, # btree.u.r.rbyd.weight,
btree.u.r.rbyd.block, # btree.u.r.rbyd.block,
btree.u.r.rbyd.trunk); # btree.u.r.rbyd.trunk);
''' #'''
#
[cases.test_btree_reinline_pop_pop_push] #[cases.test_btree_reinline_pop_pop_push]
in = 'lfs.c' #in = 'lfs.c'
defines.SHIFT = 'range(5)' #defines.SHIFT = 'range(5)'
code = ''' #code = '''
lfs_t lfs; # lfs_t lfs;
lfs_init(&lfs, CFG) => 0; # lfs_init(&lfs, CFG) => 0;
// create free lookahead # // create free lookahead
memset(lfs.lookahead.buffer, 0, CFG->lookahead_size); # memset(lfs.lookahead.buffer, 0, CFG->lookahead_size);
lfs.lookahead.start = 0; # lfs.lookahead.start = 0;
lfs.lookahead.size = lfs_min(8*CFG->lookahead_size, # lfs.lookahead.size = lfs_min(8*CFG->lookahead_size,
CFG->block_count); # CFG->block_count);
lfs.lookahead.next = 0; # lfs.lookahead.next = 0;
lfs_alloc_ack(&lfs); # lfs_alloc_ack(&lfs);
#
// create an uninlined tree # // create an uninlined tree
lfsr_btree_t btree = LFSR_BTREE_NULL; # lfsr_btree_t btree = LFSR_BTREE_NULL;
lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1,
LFSR_DATA("a", 1)) => 0; # LFSR_DATA("a", 1)) => 0;
lfsr_btree_push(&lfs, &btree, 1, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, 1, LFSR_TAG_INLINED, 1,
LFSR_DATA("b", 1)) => 0; # LFSR_DATA("b", 1)) => 0;
assert(lfsr_btree_weight(&btree) == 2); # assert(lfsr_btree_weight(&btree) == 2);
assert(!lfsr_btree_isinlined(&btree)); # assert(!lfsr_btree_isinlined(&btree));
#
// pop! our btree should now be reinlinable # // pop! our btree should now be reinlinable
lfsr_btree_pop(&lfs, &btree, 0) => 0; # lfsr_btree_pop(&lfs, &btree, 0) => 0;
#
// It's difficult to test reinlining during push or pop, since we can't just # // It's difficult to test reinlining during push or pop, since we can't just
// repeat the action until compaction occurs. # // repeat the action until compaction occurs.
// # //
// What we do here is alternate between 0 and 1 entries, eventually we # // What we do here is alternate between 0 and 1 entries, eventually we
// will compact during one of either a push or pop. To try to cover both, # // will compact during one of either a push or pop. To try to cover both,
// test with some number of extra commits to hopefully adjust where the # // test with some number of extra commits to hopefully adjust where the
// compaction ends up. # // compaction ends up.
for (lfs_size_t i = 0; i < SHIFT; i++) { # for (lfs_size_t i = 0; i < SHIFT; i++) {
lfsr_btree_set(&lfs, &btree, 0, LFSR_TAG_INLINED, 1, # lfsr_btree_set(&lfs, &btree, 0, LFSR_TAG_INLINED, 1,
LFSR_DATA("b", 1)) => 0; # LFSR_DATA("b", 1)) => 0;
} # }
#
// alternate between push/pop until compaction occurs # // alternate between push/pop until compaction occurs
lfs_block_t before_block = btree.u.r.rbyd.block; # lfs_block_t before_block = btree.u.r.rbyd.block;
for (lfs_block_t i = 0;; i++) { # for (lfs_block_t i = 0;; i++) {
// a bit hacky, but this catches infinite loops # // a bit hacky, but this catches infinite loops
assert(i < BLOCK_SIZE); # assert(i < BLOCK_SIZE);
#
// pop! # // pop!
lfsr_btree_pop(&lfs, &btree, 0) => 0; # lfsr_btree_pop(&lfs, &btree, 0) => 0;
#
assert(lfsr_btree_weight(&btree) == 0); # assert(lfsr_btree_weight(&btree) == 0);
#
// inlined? consider this a success # // inlined? consider this a success
if (lfsr_btree_isinlined(&btree)) { # if (lfsr_btree_isinlined(&btree)) {
break; # break;
} # }
#
// assert if a compaction occurred that wasn't inlined # // assert if a compaction occurred that wasn't inlined
assert(btree.u.r.rbyd.block == before_block); # assert(btree.u.r.rbyd.block == before_block);
#
// push! # // push!
lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1,
LFSR_DATA("c", 1)) => 0; # LFSR_DATA("c", 1)) => 0;
#
// try looking up tag to hopefully catch if something breaks # // try looking up tag to hopefully catch if something breaks
uint8_t buffer[4]; # uint8_t buffer[4];
lfsr_tag_t tag_; # lfsr_tag_t tag_;
lfs_size_t weight_; # lfs_size_t weight_;
#
lfsr_btree_get(&lfs, &btree, 0, # lfsr_btree_get(&lfs, &btree, 0,
&tag_, &weight_, buffer, 4) => 1; # &tag_, &weight_, buffer, 4) => 1;
assert(tag_ == LFSR_TAG_INLINED); # assert(tag_ == LFSR_TAG_INLINED);
assert(weight_ == 1); # assert(weight_ == 1);
assert(memcmp(buffer, "c", 1) == 0); # assert(memcmp(buffer, "c", 1) == 0);
#
// inlined? consider this a success # // inlined? consider this a success
if (lfsr_btree_isinlined(&btree)) { # if (lfsr_btree_isinlined(&btree)) {
break; # break;
} # }
#
// assert if a compaction occurred that wasn't inlined # // assert if a compaction occurred that wasn't inlined
assert(btree.u.r.rbyd.block == before_block); # assert(btree.u.r.rbyd.block == before_block);
} # }
#
printf("btree: w%d 0x%x.%x\n", # printf("btree: w%d 0x%x.%x\n",
btree.u.r.rbyd.weight, # btree.u.r.rbyd.weight,
btree.u.r.rbyd.block, # btree.u.r.rbyd.block,
btree.u.r.rbyd.trunk); # btree.u.r.rbyd.trunk);
''' #'''
#
[cases.test_btree_reinline_pop_push] #[cases.test_btree_reinline_pop_push]
in = 'lfs.c' #in = 'lfs.c'
defines.SIBLING = [0, 1] #defines.SIBLING = [0, 1]
defines.SHIFT = 'range(5)' #defines.SHIFT = 'range(5)'
code = ''' #code = '''
lfs_t lfs; # lfs_t lfs;
lfs_init(&lfs, CFG) => 0; # lfs_init(&lfs, CFG) => 0;
// create free lookahead # // create free lookahead
memset(lfs.lookahead.buffer, 0, CFG->lookahead_size); # memset(lfs.lookahead.buffer, 0, CFG->lookahead_size);
lfs.lookahead.start = 0; # lfs.lookahead.start = 0;
lfs.lookahead.size = lfs_min(8*CFG->lookahead_size, # lfs.lookahead.size = lfs_min(8*CFG->lookahead_size,
CFG->block_count); # CFG->block_count);
lfs.lookahead.next = 0; # lfs.lookahead.next = 0;
lfs_alloc_ack(&lfs); # lfs_alloc_ack(&lfs);
#
// create an uninlined tree # // create an uninlined tree
lfsr_btree_t btree = LFSR_BTREE_NULL; # lfsr_btree_t btree = LFSR_BTREE_NULL;
lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, 0, LFSR_TAG_INLINED, 1,
LFSR_DATA("a", 1)) => 0; # LFSR_DATA("a", 1)) => 0;
lfsr_btree_push(&lfs, &btree, 1, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, 1, LFSR_TAG_INLINED, 1,
LFSR_DATA("b", 1)) => 0; # LFSR_DATA("b", 1)) => 0;
assert(lfsr_btree_weight(&btree) == 2); # assert(lfsr_btree_weight(&btree) == 2);
assert(!lfsr_btree_isinlined(&btree)); # assert(!lfsr_btree_isinlined(&btree));
#
// It's difficult to test reinlining during push or pop, since we can't just # // It's difficult to test reinlining during push or pop, since we can't just
// repeat the action until compaction occurs. # // repeat the action until compaction occurs.
// # //
// Here we alternate between 1 and 2 entries, with the hope that compaction # // Here we alternate between 1 and 2 entries, with the hope that compaction
// occurs on the pop. We try this with some number of extra commits to make # // occurs on the pop. We try this with some number of extra commits to make
// it more likely pop is tested. # // it more likely pop is tested.
// # //
// It's possible our commits line up so compaction always occurs on a push! # // It's possible our commits line up so compaction always occurs on a push!
// For this reason, we end the test if an non-reinlining compaction occurs. # // For this reason, we end the test if an non-reinlining compaction occurs.
for (lfs_size_t i = 0; i < SHIFT; i++) { # for (lfs_size_t i = 0; i < SHIFT; i++) {
lfsr_btree_set(&lfs, &btree, 0, LFSR_TAG_INLINED, 1, # lfsr_btree_set(&lfs, &btree, 0, LFSR_TAG_INLINED, 1,
LFSR_DATA("a", 1)) => 0; # LFSR_DATA("a", 1)) => 0;
} # }
#
// alternate between push/pop until compaction occurs # // alternate between push/pop until compaction occurs
lfs_block_t before_block = btree.u.r.rbyd.block; # lfs_block_t before_block = btree.u.r.rbyd.block;
for (lfs_block_t i = 0;; i++) { # for (lfs_block_t i = 0;; i++) {
// a bit hacky, but this catches infinite loops # // a bit hacky, but this catches infinite loops
assert(i < BLOCK_SIZE); # assert(i < BLOCK_SIZE);
#
// pop! # // pop!
lfsr_btree_pop(&lfs, &btree, SIBLING) => 0; # lfsr_btree_pop(&lfs, &btree, SIBLING) => 0;
#
assert(lfsr_btree_weight(&btree) == 1); # assert(lfsr_btree_weight(&btree) == 1);
#
// try looking up tag to hopefully catch if something breaks # // try looking up tag to hopefully catch if something breaks
uint8_t buffer[4]; # uint8_t buffer[4];
lfsr_tag_t tag_; # lfsr_tag_t tag_;
lfs_size_t weight_; # lfs_size_t weight_;
#
lfsr_btree_get(&lfs, &btree, 0, # lfsr_btree_get(&lfs, &btree, 0,
&tag_, &weight_, buffer, 4) => 1; # &tag_, &weight_, buffer, 4) => 1;
assert(tag_ == LFSR_TAG_INLINED); # assert(tag_ == LFSR_TAG_INLINED);
assert(weight_ == 1); # assert(weight_ == 1);
assert(memcmp(buffer, (SIBLING == 1 ? "a" : "b"), 1) == 0); # assert(memcmp(buffer, (SIBLING == 1 ? "a" : "b"), 1) == 0);
#
// inlined? consider this a success # // inlined? consider this a success
if (lfsr_btree_isinlined(&btree)) { # if (lfsr_btree_isinlined(&btree)) {
break; # break;
} # }
#
// abort if a compaction occurs # // abort if a compaction occurs
if (btree.u.r.rbyd.block != before_block) { # if (btree.u.r.rbyd.block != before_block) {
break; # break;
} # }
#
// push! # // push!
lfsr_btree_push(&lfs, &btree, SIBLING, LFSR_TAG_INLINED, 1, # lfsr_btree_push(&lfs, &btree, SIBLING, LFSR_TAG_INLINED, 1,
LFSR_DATA("c", 1)) => 0; # LFSR_DATA("c", 1)) => 0;
#
// try looking up tag to hopefully catch if something breaks # // try looking up tag to hopefully catch if something breaks
lfsr_btree_get(&lfs, &btree, 0, # lfsr_btree_get(&lfs, &btree, 0,
&tag_, &weight_, buffer, 4) => 1; # &tag_, &weight_, buffer, 4) => 1;
assert(tag_ == LFSR_TAG_INLINED); # assert(tag_ == LFSR_TAG_INLINED);
assert(weight_ == 1); # assert(weight_ == 1);
assert(memcmp(buffer, (SIBLING == 1 ? "a" : "c"), 1) == 0); # assert(memcmp(buffer, (SIBLING == 1 ? "a" : "c"), 1) == 0);
#
lfsr_btree_get(&lfs, &btree, 1, # lfsr_btree_get(&lfs, &btree, 1,
&tag_, &weight_, buffer, 4) => 1; # &tag_, &weight_, buffer, 4) => 1;
assert(tag_ == LFSR_TAG_INLINED); # assert(tag_ == LFSR_TAG_INLINED);
assert(weight_ == 1); # assert(weight_ == 1);
assert(memcmp(buffer, (SIBLING == 1 ? "c" : "b"), 1) == 0); # assert(memcmp(buffer, (SIBLING == 1 ? "c" : "b"), 1) == 0);
#
// inlined? consider this a success # // inlined? consider this a success
if (lfsr_btree_isinlined(&btree)) { # if (lfsr_btree_isinlined(&btree)) {
break; # break;
} # }
#
// abort if a compaction occurs # // abort if a compaction occurs
if (btree.u.r.rbyd.block != before_block) { # if (btree.u.r.rbyd.block != before_block) {
break; # break;
} # }
} # }
#
printf("btree: w%d 0x%x.%x\n", # printf("btree: w%d 0x%x.%x\n",
btree.u.r.rbyd.weight, # btree.u.r.rbyd.weight,
btree.u.r.rbyd.block, # btree.u.r.rbyd.block,
btree.u.r.rbyd.trunk); # btree.u.r.rbyd.trunk);
''' #'''
# Some more general fuzz testing # Some more general fuzz testing