Rough implementation of ckreads
With the adoption of the odd-parity-zero rbyd perturb scheme, it's now possible to validate individual tag's parity with neighboring valid bits. This sparked an idea that I previously thought was intractable. If we: 1. Validate all metadata reads by checking their on-disk parity bits. 2. Validate all data reads by checking their in-metadata checksums. We end up with a closed system where all reads are checked by at least a parity bit. Being able to check all reads is a very valuable filesystem feature, but difficult for littlefs: - We need to keep relevant data in RAM while validating checksums. We can't just validate checksums and then perform a second read as that creates a hole where new bit-errors may be introduced. - This is solved in other filesystems by loading and checking whole blocks in RAM. We just can't do that here. - Without parity, we would need to check the rbyd's checksum on every tag read. This would lead to a crazy O(n^2 log n) rbyd compaction runtime. Which is why I original thought ckreads was just intractable. Now, this isn't all sunshine and rainbows. ckreads, as implemented here, has some deeply concerning flaws: - A parity bit is, mathematically, the minimum possible error-detection possible. Is validating reads with only a parity bit sufficient for real world applications? - Validating data checksums on every read may have severe performance implications. We need to read up to the entire block, which can lead to O(n^2) behavior when performing a lot of small reads in a file. - In order to validate checksums/parity-bits, we need to know where the checksums/parity-bits actually are for each piece of data. Our lfsr_data_t struct provides a surprisingly nice abstraction for this, but oof is it expensive. For the added code/stack cost alone, we probably want to eventually make this an opt-in compile-time feature. --- Implementation notes: - This found an actual compiler bug! Turns out increasing lfsr_data_t from 3-words to 5-words confuses GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101854 - Mid-commit, we may have not actually written the last tag's parity yet, which is a bit of a problem because we may read the last tag when building the next trunk! Fixing this required a whole separate tailck mechanism, which just tracks in-progress commit's parity bits. This doesn't help the code/stack cost situation... - lfsr_bd_read/cmp/cpy all need to be extended to support calculating a checksum on the side, which is a bit of a mess. - bptr's cksize/cksum is redundant now, which is going to make conditional compilation a mess. - The extra parity byte we need to read makes hint calculation a pain. Code cost wise... yeah, it's significant. Turns out almost doubling lfsr_data_t has a significant impact on stack usage. Add in all the extra code to track checksums/parity-bits and validate checksums/ parity-bits and you got yourself a pretty heavy feature: code stack before: 36352 2672 after: 38100 (+4.8%) 3032 (+13.5%)
This commit is contained in:
Christopher Haster