Github/littlefs
1
Fork 0
Code Actions Activity

Added bptr checksums

Browse Source 
Looking forward, bptr checksums provide an easy mechanism to validate
data residing in blocks. This extends the merkle-tree-like nature of the
filesystem all the way down to the data level, and is common in other
COW filesystems.

Two interesting things to note:

1. We don't actually check data-level checksums yet, but we do calculate
   data-level checksums unconditionally.

   Writing checksums is easy, but validating checksums is a bit more
   tricky. This is made a bit harder for littlefs, since we can't hold
   an entire block of data in RAM, so we have to choose between separate
   bus transactions for checksum + data reads, or extremely expensive
   overreads every read.

   Note this already exists at the metadata-level, the separate bus
   transactions for rbyd fetch + rbyd lookup means we _are_ susceptible
   to a very small window where bit errors can get through.

   But anyways, writing checksums is easy. And has basically no cost
   since we are already processing the data for our write. So we might
   as well write the data-level checksums at all times, even if we
   aren't validating at the data-level.

2. To make bptr checksums work cheaply we need an additional cksize
   field to indicate how much data is checksummed.

   This field seems redundant when we already have the bptr's data size,
   but if we didn't have this field, we would be forced to recalculate
   the checksum every time a block is sliced. This would be
   unreasonable.

   The immutable cksize field does mean we may be checksumming more data
   than we need to when validating, but we should be avoiding small
   block slices anyways for storage cost reasons.

This does add some stack cost because our bptr struct is larger now:

            code          stack
  before:  31200           2768
  after:   31272 (+0.2%)   2800 (+1.1%)
This commit is contained in:
Christopher Haster
2023-12-10 13:53:38 -06:00
parent 16fa88aac3
commit c4d75efa40
5 changed files with 202 additions and 178 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/dbgrbyd.py
+14 -18
View File
@@ -549,7 +549,7 @@ def dbg_log(data, block_size, rev, eoff, weight, *,
rid = lower_ + w-1
# show human-readable tag representation
print('%s%08x:%s %*s%s%*s %-*s%s%s' % (
print('%s%08x:%s %*s%s%*s %-*s%s%s%s' % (
'\x1b[90m' if color and j >= eoff else '',
j,
'\x1b[m' if color and j >= eoff else '',
@@ -558,17 +558,15 @@ def dbg_log(data, block_size, rev, eoff, weight, *,
2*w_width+1, '' if (tag & 0xe000) != 0x0000
else '%d-%d' % (rid-(w-1), rid) if w > 1
else rid,
56+w_width, '%-*s%s' % (
56+w_width, '%-*s %s' % (
21+w_width, tagrepr(tag, w, size, j),
' %s' % next(xxd(
data[j+d:j+d+min(size, 8)], 8), '')
next(xxd(data[j+d:j+d+min(size, 8)], 8), '')
if not args.get('raw') and not args.get('no_truncate')
and not tag & TAG_ALT else ''),
' (%s)' % ', '.join(notes) if notes else '',
'\x1b[m' if color and j >= eoff else '',
' (%s)' % ', '.join(notes) if notes
else ' %s' % jumprepr(j)
if args.get('jumps')
else ''))
' %s' % jumprepr(j)
if args.get('jumps') and not notes else ''))
# show in-device representation, including some extra
# cksum/parity info
@@ -822,18 +820,16 @@ def dbg_tree(data, block_size, rev, trunk, weight, *,
break
# show human-readable tag representation
print('%08x: %s%s' % (
print('%08x: %s%*s %-*s %s' % (
j,
treerepr(rid, tag) if args.get('tree') else '',
'%*s %-*s%s' % (
2*w_width+1, '%d-%d' % (rid-(w-1), rid)
if w > 1 else rid
if w > 0 or i == 0 else '',
21+w_width, tagrepr(tag, w, size, j),
' %s' % next(xxd(
data[j+d:j+d+min(size, 8)], 8), '')
if not args.get('raw') and not args.get('no_truncate')
and not tag & TAG_ALT else '')))
2*w_width+1, '%d-%d' % (rid-(w-1), rid)
if w > 1 else rid
if w > 0 or i == 0 else '',
21+w_width, tagrepr(tag, w, size, j),
next(xxd(data[j+d:j+d+min(size, 8)], 8), '')
if not args.get('raw') and not args.get('no_truncate')
and not tag & TAG_ALT else ''))
# show in-device representation
if args.get('device'):