Rough implementation of ckreads
With the adoption of the odd-parity-zero rbyd perturb scheme, it's now possible to validate individual tag's parity with neighboring valid bits. This sparked an idea that I previously thought was intractable. If we: 1. Validate all metadata reads by checking their on-disk parity bits. 2. Validate all data reads by checking their in-metadata checksums. We end up with a closed system where all reads are checked by at least a parity bit. Being able to check all reads is a very valuable filesystem feature, but difficult for littlefs: - We need to keep relevant data in RAM while validating checksums. We can't just validate checksums and then perform a second read as that creates a hole where new bit-errors may be introduced. - This is solved in other filesystems by loading and checking whole blocks in RAM. We just can't do that here. - Without parity, we would need to check the rbyd's checksum on every tag read. This would lead to a crazy O(n^2 log n) rbyd compaction runtime. Which is why I original thought ckreads was just intractable. Now, this isn't all sunshine and rainbows. ckreads, as implemented here, has some deeply concerning flaws: - A parity bit is, mathematically, the minimum possible error-detection possible. Is validating reads with only a parity bit sufficient for real world applications? - Validating data checksums on every read may have severe performance implications. We need to read up to the entire block, which can lead to O(n^2) behavior when performing a lot of small reads in a file. - In order to validate checksums/parity-bits, we need to know where the checksums/parity-bits actually are for each piece of data. Our lfsr_data_t struct provides a surprisingly nice abstraction for this, but oof is it expensive. For the added code/stack cost alone, we probably want to eventually make this an opt-in compile-time feature. --- Implementation notes: - This found an actual compiler bug! Turns out increasing lfsr_data_t from 3-words to 5-words confuses GCC: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101854 - Mid-commit, we may have not actually written the last tag's parity yet, which is a bit of a problem because we may read the last tag when building the next trunk! Fixing this required a whole separate tailck mechanism, which just tracks in-progress commit's parity bits. This doesn't help the code/stack cost situation... - lfsr_bd_read/cmp/cpy all need to be extended to support calculating a checksum on the side, which is a bit of a mess. - bptr's cksize/cksum is redundant now, which is going to make conditional compilation a mess. - The extra parity byte we need to read makes hint calculation a pain. Code cost wise... yeah, it's significant. Turns out almost doubling lfsr_data_t has a significant impact on stack usage. Add in all the extra code to track checksums/parity-bits and validate checksums/ parity-bits and you got yourself a pretty heavy feature: code stack before: 36352 2672 after: 38100 (+4.8%) 3032 (+13.5%)
This commit is contained in:
+29
-29
@@ -2815,7 +2815,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// test all permutations of a given size
|
||||
@@ -3140,7 +3140,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try removing each tag
|
||||
@@ -3290,7 +3290,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try removing each tag
|
||||
@@ -3905,7 +3905,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// test all permutations of a given size
|
||||
@@ -6054,7 +6054,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// test all permutations of a given size
|
||||
@@ -6221,7 +6221,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try removing each tag
|
||||
@@ -6401,7 +6401,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// test all permutations of a given size
|
||||
@@ -10615,7 +10615,7 @@ code = '''
|
||||
// copy block so we can reset after each delete
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try deleting each rid
|
||||
@@ -10784,7 +10784,7 @@ code = '''
|
||||
// copy block so we can reset after each delete
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try deleting each rid
|
||||
@@ -10977,7 +10977,7 @@ code = '''
|
||||
// copy block so we can reset after each delete
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try deleting each rid
|
||||
@@ -11114,7 +11114,7 @@ code = '''
|
||||
// copy block so we can reset after each delete
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try deleting each rid
|
||||
@@ -11547,7 +11547,7 @@ code = '''
|
||||
// copy block so we can reset after each delete
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// test all permutations of a given size
|
||||
@@ -11703,7 +11703,7 @@ code = '''
|
||||
// copy block so we can reset after each delete
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// test all permutations of a given size
|
||||
@@ -13808,7 +13808,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try growing each rid
|
||||
@@ -13940,7 +13940,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try growing each rid
|
||||
@@ -14084,7 +14084,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try growing each rid
|
||||
@@ -14241,7 +14241,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try shrinking each rid
|
||||
@@ -14373,7 +14373,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try shrinking each rid
|
||||
@@ -14517,7 +14517,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try shrinking each rid
|
||||
@@ -14673,7 +14673,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try deleting each rid
|
||||
@@ -14837,7 +14837,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try appending an attr to each rid, this should not affect
|
||||
@@ -15471,7 +15471,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try removing each tag
|
||||
@@ -15615,7 +15615,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try replacing each tag
|
||||
@@ -15857,7 +15857,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try removing each tag
|
||||
@@ -16011,7 +16011,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try replacing each tag
|
||||
@@ -16248,7 +16248,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try removing each tag
|
||||
@@ -16392,7 +16392,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try replacing each tag
|
||||
@@ -16622,7 +16622,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try removing each tag
|
||||
@@ -16766,7 +16766,7 @@ code = '''
|
||||
// copy block so we can reset after each remove
|
||||
lfsr_rbyd_t backup_rbyd = rbyd;
|
||||
uint8_t *backup_block = malloc(lfsr_rbyd_eoff(&rbyd));
|
||||
lfsr_bd_read(&lfs, rbyd.blocks[0], 0, lfsr_rbyd_eoff(&rbyd),
|
||||
CFG->read(CFG, rbyd.blocks[0], 0,
|
||||
backup_block, lfsr_rbyd_eoff(&rbyd)) => 0;
|
||||
|
||||
// try replacing each tag
|
||||
|
||||
Reference in New Issue
Block a user