Changing coalesce strategy, reimplemented shrub/btree carve
Note this is already showing better code reuse, which is a good sign,
though maybe that's just the benefit of reimplementing similar logic
multiple times.
Now both reading and carving end up in the same lfsr_btree_readnext and
lfsr_btree_buildcarve functions for both btrees and shrubs. Both btrees
and shrubs are fundamentally rbyds, so we can share a lot of
functionality as long as we redirect to the correct commit function at
the last minute. This surprising opportunity for deduplication was
noticed while putting together the dbg scripts.
Planned logic (not actual function names):
lfsr_file_readnext -> lfsr_shrub_readnext
| |
| v
'---------> lfsr_btree_readnext
lfsr_file_flushbuffer -> lfsr_shrub_carve ------------.
.---------------------' |
v v
lfsr_file_flushshrub -> lfsr_btree_carve -> lfsr_btree_buildcarve
Though the btree part of the above statement is only a hypothetical at
the moment. Not even the shrubs can survive compaction now.
The reason is the new SLICE tag which needs low-level support in rbyd
compact. SLICE introduces indirect refernces to data located in the same
rbyd, which removes any copying cost associated with coalescing.
Previously, a large coalesce_size risked O(n^2) runtime when
incrementally append small amounts of data, but with SLICEs we can defer
coalescing to compaction time, where the copy is effectively free.
This compaction-time-coalescing is also hypothetical, which is why our
tests are failing. But the theory is promising.
I was originally against this idea because of how it crosses abstraction
layers, requiring some very low-level code that absolutely can not be
omitted in a simpler littlefs driver. But after working on the actual
file writing code for a while I've become convinced the tradeoff is
worth it.
Note coalesce_size will likely still need to be configurable. Data in
fragmenting/sparse btrees is still susceptible to coalescing, and it's
not clear the impacts of internal fragmentation when data sizes approach
the hard block_size/2 limit.
This commit is contained in:
+11
-9
@@ -39,15 +39,16 @@ TAG_BOOKMARK = 0x0201
|
||||
TAG_REG = 0x0202
|
||||
TAG_DIR = 0x0203
|
||||
TAG_STRUCT = 0x0300
|
||||
TAG_INLINED = 0x0300
|
||||
TAG_TRUNK = 0x0304
|
||||
TAG_BLOCK = 0x0308
|
||||
TAG_BTREE = 0x030c
|
||||
TAG_BRANCH = 0x0314
|
||||
TAG_DATA = 0x0300
|
||||
TAG_SLICE = 0x0304
|
||||
TAG_TRUNK = 0x0308
|
||||
TAG_DID = 0x030c
|
||||
TAG_BLOCK = 0x0310
|
||||
TAG_BTREE = 0x0314
|
||||
TAG_MDIR = 0x0321
|
||||
TAG_MTREE = 0x0324
|
||||
TAG_MROOT = 0x0329
|
||||
TAG_DID = 0x032c
|
||||
TAG_BRANCH = 0x032c
|
||||
TAG_UATTR = 0x0400
|
||||
TAG_SATTR = 0x0600
|
||||
TAG_SHRUB = 0x1000
|
||||
@@ -178,15 +179,16 @@ def tagrepr(tag, w, size, off=None):
|
||||
elif (tag & 0xef00) == TAG_STRUCT:
|
||||
return '%s%s%s %d' % (
|
||||
'shrub' if tag & TAG_SHRUB else '',
|
||||
'inlined' if (tag & 0xfff) == TAG_INLINED
|
||||
'data' if (tag & 0xfff) == TAG_DATA
|
||||
else 'slice' if (tag & 0xfff) == TAG_SLICE
|
||||
else 'trunk' if (tag & 0xfff) == TAG_TRUNK
|
||||
else 'did' if (tag & 0xfff) == TAG_DID
|
||||
else 'block' if (tag & 0xfff) == TAG_BLOCK
|
||||
else 'btree' if (tag & 0xfff) == TAG_BTREE
|
||||
else 'branch' if (tag & 0xfff) == TAG_BRANCH
|
||||
else 'mdir' if (tag & 0xfff) == TAG_MDIR
|
||||
else 'mtree' if (tag & 0xfff) == TAG_MTREE
|
||||
else 'mroot' if (tag & 0xfff) == TAG_MROOT
|
||||
else 'did' if (tag & 0xfff) == TAG_DID
|
||||
else 'branch' if (tag & 0xfff) == TAG_BRANCH
|
||||
else 'struct 0x%02x' % (tag & 0xff),
|
||||
' w%d' % w if w else '',
|
||||
size)
|
||||
|
||||
Reference in New Issue
Block a user