Checksumming unaligned data during block compaction is surprisingly
tricky. We don't know if our data will be aligned until after
a potentially unbounded number lookups, we need to write data into our
pcache as we go to avoid unnecessary lookups, but if we end up unaligned
we need to revert our checksum to the checksum of the aligned data.
The way I see it there are 4 options:
1. Calculate the checksum after writing data into the block.
This is the most expensive option, requiring a full second read of
the data to calculate the checksum. It is simple though.
2. Do a pass over the btree to figure out alignment before writing.
This at least only reads metadata twice, so is more efficient than
the 1st option.
3. Keep track of the aligned checksum on each flush, falling back to the
last flushed checksum if we need to correct alignment.
This solution is flexible though requires some extra state to track
multiple checksums.
4. Leverage the math behind CRCs to run the CRC backwards when we
truncate for alignment.
This works, though a bit inefficiently, but is strictly tied to
CRC-related checksums.
By inefficient I mean that we would likely be limited to a bit-level
"uncrc32c". It's possible to create nibble/byte tables for uncrc32c,
but this adds significant code cost for a relatively uncritical
function.
I was hopeful that we could leverage the existing tables in both
functions, but unfortunately it doesn't work out like that. You could
scan the crc32c table to find the constant to reverse, but this
requires ~16*2 or ~256 operations vs "naive" ~8 operations per byte.
This commit implements both 3 and 4, defaulting to 4 unless
LFS_NO_UNCRC32C is defined.
The current lfs_uncrc32c implementation is a simple bit-level
implementation, but does allow for crc32c truncation without any extra
state.
code stack
before: 32044 2880
uncrc32c: 32108 (+0.2%) 2880 (+0.0%)
flcksum: 32132 (+0.3%) 2880 (+0.0%)
Christopher Haster
006d656da2