scripts: Reworked dbglfs.py, adopted Lfs, Config, Gstate, etc

I'm starting to regret these reworks. They've been a big time sink. But
at least these should be much easier to extend with the future planned
auxiliary trees?

New classes:

- Bptr - A representation of littlefs's data-only block pointers.

  Extra fun is the lazily checked Bptr.__bool__ method, which should
  prevent slowing down scripts that don't actually verify checksums.

- Config - The set of littlefs config entries.

- Gstate - The set of littlefs gstate.

  I may have had too much fun with Config and Gstate. Not only do these
  provide lookup functions for config/gstate, but known config/gstate
  get lazily parsed classes that can provide easy access to the relevant
  metadata.

  These even abuse Python's __subclasses__, so all you need to do to add
  a new known config/gstate is extend the relevant Config.Config/
  Gstate.Gstate class.

  The __subclasses__ API is a weird but powerful one.

- Lfs - The big one, a high-level abstraction of littlefs itself.

  Contains subclasses for known files: Lfs.Reg, Lfs.Dir, Lfs.Stickynote,
  etc, which can be accessed by path, did+name, mid, etc. It even
  supports iterating over orphaned files, though it's expensive (but
  incredibly valuable for debugging!).

  Note that all file types can currently have attached bshrubs/btrees.
  In the existing implementation only reg files should actually end up
  with bshrubs/btrees, but the whole point of these scripts is to debug
  things that _shouldn't_ happen.

  I intentionally gave up on providing depth bounds in Lfs. Too
  complicated for something so high-level.

On noteworthy change is not recursing into directories by default. This
hopefully avoids overloading new users and matches the behavior of most
other Linux/Unix tools.

This adopts -r/--recurse/--file-depth for controlling how far to recurse
down directories, and -z/--depth/--tree-depth for controlling how far to
recurse down tree structures (mostly files). I like this API. It's
consistent with -z/--depth in the other dbg scripts, and -r/--recurse is
probably intuitive for most Linux/Unix users.

To make this work we did need to change -r/--raw -> -x/--raw. But --raw
is already a bit of a weird name for what really means "include a hex
dump".

Note that -z/--depth/--tree-depth does _not_ imply --files. Right now
only files can contain tree structures, but this will change when we get
around to adding the auxiliary trees.

This also adds the ability to specify a file path to use as the root
directory, though we need the leading slash to disambiguate file paths
and mroot addresses.

---

Also tagrepr has been tweaked to include the global/delta names,
toggleable with the optional global_ kwarg.

Rattr now has its own lazy parsers for did + name. A more organized
codebase would probably have a separate Name type, but it just wasn't
worth the hassle.

And the abstraction classes have all been tweaked to require the
explicit Rbyd.repr() function for a CLI-friendly representation. Relying
on __str__ hurt readability and debugging, especially since Python
prefers __str__ over __repr__ when printing things.

scripts/dbgbtree.py

@@ -6,6 +6,7 @@ if __name__ == "__main__":
 
 import bisect
 import collections as co
+import functools as ft
 import itertools as it
 import math as mt
 import os
@@ -165,12 +166,17 @@ def xxd(data, width=16):
                     b if b >= ' ' and b <= '~' else '.'
                         for b in map(chr, data[i:i+width])))
 
-def tagrepr(tag, weight=None, size=None, off=None):
+# human readable tag repr
+def tagrepr(tag, weight=None, size=None, *,
+        global_=False,
+        toff=None):
+    # null tags
     if (tag & 0x6fff) == TAG_NULL:
         return '%snull%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
                 ' w%d' % weight if weight else '',
                 ' %d' % size if size else '')
+    # config tags
     elif (tag & 0x6f00) == TAG_CONFIG:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -185,13 +191,23 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'config 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # global-state delta tags
     elif (tag & 0x6f00) == TAG_GDELTA:
-        return '%s%s%s%s' % (
+        if global_:
-                'shrub' if tag & TAG_SHRUB else '',
+            return '%s%s%s%s' % (
-                'grmdelta' if (tag & 0xfff) == TAG_GRMDELTA
+                    'shrub' if tag & TAG_SHRUB else '',
-                    else 'gdelta 0x%02x' % (tag & 0xff),
+                    'grm' if (tag & 0xfff) == TAG_GRMDELTA
-                ' w%d' % weight if weight else '',
+                        else 'gstate 0x%02x' % (tag & 0xff),
-                ' %s' % size if size is not None else '')
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+        else:
+            return '%s%s%s%s' % (
+                    'shrub' if tag & TAG_SHRUB else '',
+                    'grmdelta' if (tag & 0xfff) == TAG_GRMDELTA
+                        else 'gdelta 0x%02x' % (tag & 0xff),
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+    # name tags, includes file types
     elif (tag & 0x6f00) == TAG_NAME:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -203,6 +219,7 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'name 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # structure tags
     elif (tag & 0x6f00) == TAG_STRUCT:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -218,6 +235,7 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'struct 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # custom attributes
     elif (tag & 0x6e00) == TAG_ATTR:
         return '%s%sattr 0x%02x%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -225,37 +243,49 @@ def tagrepr(tag, weight=None, size=None, off=None):
                 ((tag & 0x100) >> 1) ^ (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # alt pointers
     elif tag & TAG_ALT:
         return 'alt%s%s 0x%03x%s%s' % (
                 'r' if tag & TAG_R else 'b',
                 'gt' if tag & TAG_GT else 'le',
                 tag & 0x0fff,
                 ' w%d' % weight if weight is not None else '',
-                ' 0x%x' % (0xffffffff & (off-size))
+                ' 0x%x' % (0xffffffff & (toff-size))
-                    if size and off is not None
+                    if size and toff is not None
                     else ' -%d' % size if size
                     else '')
+    # checksum tags
     elif (tag & 0x7f00) == TAG_CKSUM:
         return 'cksum%s%s%s%s' % (
                 'p' if not tag & 0xfe and tag & TAG_P else '',
                 ' 0x%02x' % (tag & 0xff) if tag & 0xfe else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # note tags
     elif (tag & 0x7f00) == TAG_NOTE:
         return 'note%s%s%s' % (
                 ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # erased-state checksum tags
     elif (tag & 0x7f00) == TAG_ECKSUM:
         return 'ecksum%s%s%s' % (
                 ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # global-checksum delta tags
     elif (tag & 0x7f00) == TAG_GCKSUMDELTA:
-        return 'gcksumdelta%s%s%s' % (
+        if global_:
-                ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
+            return 'gcksum%s%s%s' % (
-                ' w%d' % weight if weight else '',
+                    ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
-                ' %s' % size if size is not None else '')
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+        else:
+            return 'gcksumdelta%s%s%s' % (
+                    ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+    # unknown tags
     else:
         return '0x%04x%s%s' % (
                 tag,
@@ -302,6 +332,29 @@ class TreeBranch(co.namedtuple('TreeBranch', ['a', 'b', 'depth', 'color'])):
     def __ge__(self, other):
         return (self.depth, self.a, self.b) >= (other.depth, other.a, other.b)
 
+    # apply a function to a/b while trying to avoid copies
+    def map(self, filter_, map_=None):
+        if map_ is None:
+            filter_, map_ = None, filter_
+
+        a = self.a
+        if filter_ is None or filter_(a):
+            a = map_(a)
+
+        b = self.b
+        if filter_ is None or filter_(b):
+            b = map_(b)
+
+        if a != self.a or b != self.b:
+            return self.__class__(
+                    a if a != self.a else self.a,
+                    b if b != self.b else self.b,
+                    self.depth,
+                    self.color)
+        else:
+            return self
+
+# render some nice ascii trees
 def treerepr(tree, x, depth=None, color=False):
     # find the max depth from the tree
     if depth is None:
@@ -359,9 +412,15 @@ def pathdelta(a, b):
         a = list(a)
     i = 0
     for a_, b_ in zip(a, b):
-        if type(a_) == type(b_) and a_ == b_:
+        try:
-            i += 1
+            if type(a_) == type(b_) and a_ == b_:
-        else:
+                i += 1
+            else:
+                break
+        # treat exceptions here as failure to match, most likely
+        # the compared types are incompatible, it's the caller's
+        # problem
+        except Exception:
             break
 
     return [(i+j, a_) for j, a_ in enumerate(a[i:])]
@@ -375,17 +434,17 @@ class Bd:
         self.block_count = block_count
 
     def __repr__(self):
-        return '<%s %sx%s>' % (
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
-                self.__class__.__name__,
+
-                self.block_size,
+    def repr(self):
-                self.block_count)
+        return 'bd %sx%s' % (self.block_size, self.block_count)
 
     def read(self, size=-1):
         return self.f.read(size)
 
-    def seek(self, block, off, whence=0):
+    def seek(self, block, off=0, whence=0):
         pos = self.f.seek(block*self.block_size + off, whence)
-        return pos // block_size, pos % block_size
+        return pos // self.block_size, pos % self.block_size
 
     def readblock(self, block):
         self.f.seek(block*self.block_size)
@@ -393,22 +452,40 @@ class Bd:
 
 # tagged data in an rbyd
 class Rattr:
-    def __init__(self, tag, weight, block, toff, off, data):
+    def __init__(self, tag, weight, blocks, toff, tdata, data):
         self.tag = tag
         self.weight = weight
-        self.block = block
+        if isinstance(blocks, int):
+            self.blocks = [blocks]
+        else:
+            self.blocks = list(blocks)
         self.toff = toff
-        self.off = off
+        self.tdata = tdata
         self.data = data
 
+    @property
+    def block(self):
+        return self.blocks[0]
+
+    @property
+    def tsize(self):
+        return len(self.tdata)
+
+    @property
+    def off(self):
+        return self.toff + len(self.tdata)
+
     @property
     def size(self):
         return len(self.data)
 
-    def __repr__(self):
+    def __bytes__(self):
-        return '<%s %s>' % (self.__class__.__name__, self)
+        return self.data
 
-    def __str__(self):
+    def __repr__(self):
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
+
+    def repr(self):
         return tagrepr(self.tag, self.weight, self.size)
 
     def __iter__(self):
@@ -424,14 +501,42 @@ class Rattr:
     def __hash__(self):
         return hash((self.tag, self.weight, self.data))
 
+    # convenience for did/name access
+    def _parse_name(self):
+        # note we return a null name for non-name tags, this is so
+        # vestigial names in btree nodes act as a catch-all
+        if (self.tag & 0xff00) != TAG_NAME:
+            did = 0
+            name = b''
+        else:
+            did, d = fromleb128(self.data)
+            name = self.data[d:]
+
+        # cache both
+        self.did = did
+        self.name = name
+
+    @ft.cached_property
+    def did(self):
+        self._parse_name()
+        return self.did
+
+    @ft.cached_property
+    def name(self):
+        self._parse_name()
+        return self.name
+
 class Ralt:
-    def __init__(self, tag, weight, block, toff, off, jump,
+    def __init__(self, tag, weight, blocks, toff, tdata, jump,
             color=None, followed=None):
         self.tag = tag
         self.weight = weight
-        self.block = block
+        if isinstance(blocks, int):
+            self.blocks = [blocks]
+        else:
+            self.blocks = list(blocks)
         self.toff = toff
-        self.off = off
+        self.tdata = tdata
         self.jump = jump
 
         if color is not None:
@@ -440,15 +545,27 @@ class Ralt:
             self.color = 'r' if tag & TAG_R else 'b'
         self.followed = followed
 
+    @property
+    def block(self):
+        return self.blocks[0]
+
+    @property
+    def tsize(self):
+        return len(self.tdata)
+
+    @property
+    def off(self):
+        return self.toff + len(self.tdata)
+
     @property
     def joff(self):
         return self.toff - self.jump
 
     def __repr__(self):
-        return '<%s %s>' % (self.__class__.__name__, self)
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
 
-    def __str__(self):
+    def repr(self):
-        return tagrepr(self.tag, self.weight, self.jump, self.toff)
+        return tagrepr(self.tag, self.weight, self.jump, toff=self.toff)
 
     def __iter__(self):
         return iter((self.tag, self.weight, self.jump))
@@ -466,19 +583,20 @@ class Ralt:
 
 # our core rbyd type
 class Rbyd:
-    def __init__(self, data, blocks, trunk, weight, rev, eoff, cksum, *,
+    def __init__(self, blocks, trunk, weight, rev, eoff, cksum, data, *,
             gcksumdelta=None,
             corrupt=False):
         if isinstance(blocks, int):
-            blocks = [blocks]
+            self.blocks = [blocks]
-
+        else:
-        self.data = data
+            self.blocks = list(blocks)
-        self.blocks = list(blocks)
         self.trunk = trunk
         self.weight = weight
         self.rev = rev
         self.eoff = eoff
         self.cksum = cksum
+        self.data = data
+
         self.gcksumdelta = gcksumdelta
         self.corrupt = corrupt
 
@@ -495,10 +613,10 @@ class Rbyd:
                     self.trunk)
 
     def __repr__(self):
-        return '<%s %s w%s>' % (
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
-                self.__class__.__name__,
+
-                self.addr(),
+    def repr(self):
-                self.weight)
+        return 'rbyd %s w%s' % (self.addr(), self.weight)
 
     def __bool__(self):
         return not self.corrupt
@@ -514,11 +632,11 @@ class Rbyd:
         return hash((frozenset(self.blocks), self.trunk))
 
     @classmethod
-    def fetch(cls, bd, blocks, trunk=None, cksum=None):
+    def fetch(cls, bd, blocks, trunk=None):
         # multiple blocks? unfortunately this must be a list
         if isinstance(blocks, list):
             # fetch all blocks
-            rbyds = [cls.fetch(bd, block, trunk, cksum) for block in blocks]
+            rbyds = [cls.fetch(bd, block, trunk) for block in blocks]
             # determine most recent revision
             i = 0
             for i_, rbyd in enumerate(rbyds):
@@ -534,6 +652,9 @@ class Rbyd:
             rbyd.blocks += tuple(
                     rbyds[(i+1+j) % len(rbyds)].block
                         for j in range(len(rbyds)-1))
+            # and patch the gcksumdelta if we have one
+            if rbyd.gcksumdelta is not None:
+                rbyd.gcksumdelta.blocks = rbyd.blocks
             return rbyd
 
         block = blocks
@@ -546,9 +667,9 @@ class Rbyd:
                     else block[1] if isinstance(block, tuple)
                     else None)
 
-        # bd can be either a bd reference or preread data
+        # bd can be either a bd reference or a preread block
         #
-        # preread data can be useful for avoiding race conditions
+        # preread blocks can be useful for avoiding race conditions
         # with cksums and shrubs
         if isinstance(bd, Bd):
             # seek/read the block
@@ -558,9 +679,9 @@ class Rbyd:
 
         # fetch the rbyd
         rev = fromle32(data[0:4])
-        cksum_ = 0
+        cksum = 0
-        cksum__ = crc32c(data[0:4])
+        cksum_ = crc32c(data[0:4])
-        cksum___ = cksum__
+        cksum__ = cksum_
         perturb = False
         eoff = 0
         eoff_ = None
@@ -576,10 +697,10 @@ class Rbyd:
         while j_ < len(data) and (not trunk or eoff <= trunk):
             # read next tag
             v, tag, w, size, d = fromtag(data[j_:])
-            if v != parity(cksum___):
+            if v != parity(cksum__):
                 break
-            cksum___ ^= 0x00000080 if v else 0
+            cksum__ ^= 0x00000080 if v else 0
-            cksum___ = crc32c(data[j_:j_+d], cksum___)
+            cksum__ = crc32c(data[j_:j_+d], cksum__)
             j_ += d
             if not tag & TAG_ALT and j_ + size > len(data):
                 break
@@ -587,22 +708,23 @@ class Rbyd:
             # take care of cksums
             if not tag & TAG_ALT:
                 if (tag & 0xff00) != TAG_CKSUM:
-                    cksum___ = crc32c(data[j_:j_+size], cksum___)
+                    cksum__ = crc32c(data[j_:j_+size], cksum__)
 
                     # found a gcksumdelta?
                     if (tag & 0xff00) == TAG_GCKSUMDELTA:
-                        gcksumdelta_ = Rattr(tag, w,
+                        gcksumdelta_ = Rattr(tag, w, block, j_-d,
-                                block, j_-d, d, data[j_:j_+size])
+                                data[j_-d:j_],
+                                data[j_:j_+size])
 
                 # found a cksum?
                 else:
                     # check cksum
-                    cksum____ = fromle32(data[j_:j_+4])
+                    cksum___ = fromle32(data[j_:j_+4])
-                    if cksum___ != cksum____:
+                    if cksum__ != cksum___:
                         break
                     # commit what we have
                     eoff = eoff_ if eoff_ else j_ + size
-                    cksum_ = cksum__
+                    cksum = cksum_
                     trunk_ = trunk__
                     weight = weight_
                     gcksumdelta = gcksumdelta_
@@ -610,7 +732,7 @@ class Rbyd:
                     # update perturb bit
                     perturb = tag & TAG_P
                     # revert to data cksum and perturb
-                    cksum___ = cksum__ ^ (0xfca42daf if perturb else 0)
+                    cksum__ = cksum_ ^ (0xfca42daf if perturb else 0)
 
             # evaluate trunks
             if (tag & 0xf000) != TAG_CKSUM:
@@ -634,7 +756,7 @@ class Rbyd:
                             if trunk and j_ + size > trunk:
                                 eoff_ = j_ + size
                                 eoff = eoff_
-                                cksum_ = cksum___ ^ (
+                                cksum = cksum__ ^ (
                                         0xfca42daf if perturb else 0)
                                 trunk_ = trunk__
                                 weight = weight_
@@ -642,23 +764,34 @@ class Rbyd:
                         trunk___ = 0
 
                 # update canonical checksum, xoring out any perturb state
-                cksum__ = cksum___ ^ (0xfca42daf if perturb else 0)
+                cksum_ = cksum__ ^ (0xfca42daf if perturb else 0)
 
             if not tag & TAG_ALT:
                 j_ += size
 
-        # cksum mismatch?
+        return cls(block, trunk_, weight, rev, eoff, cksum, data,
-        if cksum is not None and cksum_ != cksum:
-            return cls(data, block, 0, 0, rev, 0, cksum_,
-                    corrupt=True)
-
-        return cls(data, block, trunk_, weight, rev, eoff, cksum_,
                 gcksumdelta=gcksumdelta,
                 corrupt=not trunk_)
 
+    @classmethod
+    def fetchck(cls, bd, blocks, trunk, weight, cksum):
+        # try to fetch the rbyd normally
+        rbyd = cls.fetch(bd, blocks, trunk)
+
+        # cksum mismatch? trunk/weight mismatch?
+        if (rbyd.cksum != cksum
+                or rbyd.trunk != trunk
+                or rbyd.weight != weight):
+            # mark as corrupt and keep track of expected trunk/weight
+            rbyd.corrupt = True
+            rbyd.trunk = trunk
+            rbyd.weight = weight
+
+        return rbyd
+
     def lookupnext(self, rid, tag=None, *,
             path=False):
-        if not self:
+        if not self or rid >= self.weight:
             return None, None, *(([],) if path else ())
 
         tag = max(tag or 0, 0x1)
@@ -694,7 +827,8 @@ class Rbyd:
                             color = 'b'
 
                         path_.append(Ralt(
-                                alt, w, self.block, j+jump, j+jump+d, jump,
+                                alt, w, self.blocks, j+jump,
+                                self.data[j+jump:j+jump+d], jump,
                                 color=color,
                                 followed=True))
 
@@ -716,7 +850,8 @@ class Rbyd:
                             color = 'b'
 
                         path_.append(Ralt(
-                                alt, w, self.block, j-d, j, jump,
+                                alt, w, self.blocks, j-d,
+                                self.data[j-d:j], jump,
                                 color=color,
                                 followed=False))
 
@@ -730,7 +865,8 @@ class Rbyd:
                     return None, None, *(([],) if path else ())
 
                 return (rid_,
-                        Rattr(tag_, w_, self.block, j, j+d,
+                        Rattr(tag_, w_, self.blocks, j,
+                            self.data[j:j+d],
                             self.data[j+d:j+d+jump]),
                         *((path_,) if path else ()))
 
@@ -784,7 +920,7 @@ class Rbyd:
             yield rid, name, *path_
             rid += 1
 
-    def rattrs_(self, rid=None, *,
+    def rattrs_(self, rid=None, tag=None, mask=None, *,
             path=False):
         if rid is None:
             rid, tag = -1, 0
@@ -798,24 +934,31 @@ class Rbyd:
                 yield rid, rattr, *path_
                 tag = rattr.tag
         else:
-            tag = 0
+            if tag is None:
+                tag, mask = 0, 0xffff
+            if mask is None:
+                mask = 0
+
+            tag_ = max((tag & ~mask) - 1, 0)
             while True:
-                rid_, rattr, *path_ = self.lookupnext(rid, tag+0x1,
+                rid_, rattr_, *path_ = self.lookupnext(rid, tag_+0x1,
                         path=path)
                 # found end of tree?
-                if rid_ is None or rid_ != rid:
+                if (rid_ is None
+                        or rid_ != rid
+                        or (rattr_.tag & ~mask) != (tag & ~mask)):
                     break
 
-                yield rattr, *path_
+                yield rattr_, *path_
-                tag = rattr.tag
+                tag_ = rattr_.tag
 
-    def rattrs(self, rid=None, *,
+    def rattrs(self, rid=None, tag=None, mask=None, *,
             path=False):
         if rid is None:
-            yield from self.rattrs_(rid,
+            yield from self.rattrs_(rid, tag, mask,
                     path=path)
         else:
-            for rattr, *path_ in self.rattrs_(rid,
+            for rattr, *path_ in self.rattrs_(rid, tag, mask,
                     path=path):
                 if path:
                     yield rattr, *path_
@@ -828,33 +971,25 @@ class Rbyd:
     # lookup by name
     def namelookup(self, did, name):
         # binary search
-        best = (False, None, None, None, None)
+        best = None, None
         lower = 0
         upper = self.weight
         while lower < upper:
-            rid, rattr = self.lookupnext(lower + (upper-1-lower)//2)
+            rid, name_ = self.lookupnext(
+                lower + (upper-1-lower)//2)
             if rid is None:
                 break
 
-            # treat vestigial names as a catch-all
-            if ((rattr.tag == TAG_NAME and rid-(rattr.weight-1) == 0)
-                    or (rattr.tag & 0xff00) != TAG_NAME):
-                did_ = 0
-                name_ = b''
-            else:
-                did_, d = fromleb128(rattr.data)
-                name_ = rattr.data[d:]
-
             # bisect search space
-            if (did_, name_) > (did, name):
+            if (name_.did, name_.name) > (did, name):
-                upper = rid-(w-1)
+                upper = rid-(name_.weight-1)
-            elif (did_, name_) < (did, name):
+            elif (name_.did, name_.name) < (did, name):
                 lower = rid + 1
                 # keep track of best match
-                best = (False, rid, rattr)
+                best = rid, name_
             else:
                 # found a match
-                return True, rid, rattr
+                return rid, name_
 
         return best
 
@@ -1006,10 +1141,10 @@ class Btree:
         return self.rbyd.addr()
 
     def __repr__(self):
-        return '<%s %s w%s>' % (
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
-                self.__class__.__name__,
+
-                self.addr(),
+    def repr(self):
-                self.weight)
+        return 'btree %s w%s' % (self.addr(), self.weight)
 
     def __eq__(self, other):
         return self.rbyd == other.rbyd
@@ -1021,16 +1156,40 @@ class Btree:
         return hash(self.rbyd)
 
     @classmethod
-    def fetch(cls, bd, blocks, trunk=None, cksum=None):
+    def fetch(cls, bd, blocks, trunk=None):
-        # we need a real bd reference here
+        # bd can either be a bd reference or a tuple of bd + data to
+        # avoid rereads, but we need a real bd reference somehow
+        if isinstance(bd, tuple):
+            bd, data = bd
+        else:
+            bd, data = bd, bd
         assert isinstance(bd, Bd)
 
-        rbyd = Rbyd.fetch(bd, blocks, trunk, cksum)
+        # rbyd fetch does most of the work here
+        rbyd = Rbyd.fetch(data, blocks, trunk)
+        return cls(bd, rbyd)
+
+    @classmethod
+    def fetchck(cls, bd, blocks, trunk, weight, cksum):
+        # bd can either be a bd reference or a tuple of bd + data to
+        # avoid rereads, but we need a real bd reference somehow
+        if isinstance(bd, tuple):
+            bd, data = bd
+        else:
+            bd, data = bd, bd
+        assert isinstance(bd, Bd)
+
+        # rbyd fetchck does most of the work here
+        rbyd = Rbyd.fetchck(data, blocks, trunk, weight, cksum)
         return cls(bd, rbyd)
 
     def lookupleaf(self, bid, *,
             path=None,
             depth=None):
+        if not self or bid >= self.weight:
+            return (None, None, None, None,
+                    *(([],) if path else ()))
+
         rbyd = self.rbyd
         rid = bid
         depth_ = 1
@@ -1059,11 +1218,8 @@ class Btree:
             if branch_ is not None and (
                     not depth or depth_ < depth):
                 block, trunk, cksum = frombranch(branch_.data)
-                rbyd = Rbyd.fetch(self.bd, block, trunk, cksum)
+                rbyd = Rbyd.fetchck(self.bd, block, trunk, name_.weight,
-                # keep track of expected trunk/weight if corrupted
+                        cksum)
-                if not rbyd:
-                    rbyd.trunk = trunk
-                    rbyd.weight = name_.weight
 
                 rid -= (rid_-(name_.weight-1))
                 depth_ += 1
@@ -1072,22 +1228,51 @@ class Btree:
                 return (bid + (rid_-rid), rbyd, rid_, name_,
                         *((path_,) if path else ()))
 
-    def lookup(self, bid, tag=None, mask=None, *,
+    # the non-leaf variants discard the rbyd info, these can be a bit
+    # more convenient, but at a performance cost
+    def lookupnext(self, bid, *,
+            path=None,
+            depth=None):
+        # just discard the rbyd info
+        bid, rbyd, rid, name, *path_ = self.lookupleaf(bid,
+                path=path,
+                depth=depth)
+        return bid, name, *path_
+
+    def lookup_(self, bid, tag=None, mask=None, *,
             path=False,
             depth=None):
         # lookup rbyd in btree
+        #
+        # note this function expects bid to be known, use lookupnext
+        # first if you don't care about the exact bid (or better yet,
+        # lookupleaf and call lookup on the returned rbyd)
+        #
+        # this matches rbyd's lookup behavior, which needs a known rid
+        # to avoid a double lookup
         bid_, rbyd_, rid_, name_, *path_ = self.lookupleaf(bid,
                 path=path,
                 depth=depth)
-        if bid_ is None:
+        if bid_ is None or bid_ != bid:
-            return None, None, None, None, None, *path_
+            return None, *path_
 
         # lookup tag in rbyd
         rattr_ = rbyd_.lookup(rid_, tag, mask)
         if rattr_ is None:
-            return None, None, None, None, None, *path_
+            return None, *path_
 
-        return bid_, rbyd_, rid_, name_, rattr_, *path_
+        return rattr_, *path_
+
+    def lookup(self, bid, tag=None, mask=None, *,
+            path=False,
+            depth=None):
+        rattr, *path_ = self.lookup_(bid, tag, mask,
+                path=path,
+                depth=depth)
+        if path:
+            return rattr, *path_
+        else:
+            return rattr
 
     def __getitem__(self, key):
         if not isinstance(key, tuple):
@@ -1099,7 +1284,7 @@ class Btree:
         if not isinstance(key, tuple):
             key = (key,)
 
-        return self.lookup(*key)[0] is not None
+        return self.lookup_(*key)[0] is not None
 
     # note leaves only iterates over leaf rbyds, whereas traverse
     # traverses all rbyds
@@ -1108,7 +1293,8 @@ class Btree:
             depth=None):
         # include our root rbyd even if the weight is zero
         if self.weight == 0:
-            yield -1, self.rbyd, *(([],) if path else())
+            yield -1, self.rbyd, *(([],) if path else ())
+            return
 
         bid = 0
         while True:
@@ -1139,21 +1325,27 @@ class Btree:
                 yield bid_, rbyd_, *((path_[:d],) if path else ())
             ptrunk_ = trunk_
 
+    # note bids/rattrs do _not_ include corrupt btree nodes!
     def bids(self, *,
+            leaves=False,
             path=False,
             depth=None):
         for bid, rbyd, *path_ in self.leaves(
                 path=path,
                 depth=depth):
             for rid, name in rbyd.rids():
-                yield (bid-(rbyd.weight-1) + rid,
+                bid_ = bid-(rbyd.weight-1) + rid
-                        rbyd, rid, name,
+                if leaves:
-                        *((path_[0]+[
+                    yield (bid_, rbyd, rid, name,
-                                (bid-(rbyd.weight-1) + rid,
+                            *((path_[0]+[(bid_, rbyd, rid, name)],)
-                                    rbyd, rid, name)],)
+                                if path else ()))
-                            if path else ()))
+                else:
+                    yield (bid_, name,
+                            *((path_[0]+[(bid_, rbyd, rid, name)],)
+                                if path else ()))
 
-    def rattrs_(self, bid=None, *,
+    def rattrs_(self, bid=None, tag=None, mask=None, *,
+            leaves=False,
             path=False,
             depth=None):
         if bid is None:
@@ -1161,48 +1353,68 @@ class Btree:
                     path=path,
                     depth=depth):
                 for rid, name in rbyd.rids():
+                    bid_ = bid-(rbyd.weight-1) + rid
                     for rattr in rbyd.rattrs(rid):
-                        yield (bid-(rbyd.weight-1) + rid,
+                        if leaves:
-                                rbyd, rid, name, rattr,
+                            yield (bid_, rbyd, rid, rattr,
-                                *((path_[0]+[
+                                    *((path_[0]+[(bid_, rbyd, rid, name)],)
-                                        (bid-(rbyd.weight-1) + rid,
+                                        if path else ()))
-                                            rbyd, rid, name)],)
+                        else:
-                                    if path else ()))
+                            yield (bid_, rattr,
+                                    *((path_[0]+[(bid_, rbyd, rid, name)],)
+                                        if path else ()))
         else:
             bid, rbyd, rid, name, *path_ = self.lookupleaf(bid,
                     path=path,
                     depth=depth)
-            for rattr in rbyd.rattrs(rid):
+            if bid is None:
-                yield rattr, *path_
+                return
 
-    def rattrs(self, bid=None, *,
+            for rattr in rbyd.rattrs(rid, tag, mask):
+                if leaves:
+                    yield rbyd, rid, rattr, *path_
+                else:
+                    yield rattr, *path_
+
+    def rattrs(self, bid=None, tag=None, mask=None, *,
+            leaves=False,
             path=False,
             depth=None):
-        if bid is None:
+        if bid is None or leaves or path:
-            yield from self.rattrs_(bid,
+            yield from self.rattrs_(bid, tag, mask,
+                    leaves=leaves,
                     path=path,
                     depth=depth)
         else:
-            for rattr, *path_ in self.rattrs_(bid,
+            for rattr, *path_ in self.rattrs_(bid, tag, mask,
+                    leaves=leaves,
                     path=path,
                     depth=depth):
-                if path:
+                yield rattr
-                    yield rattr, *path_
-                else:
-                    yield rattr
 
     def __iter__(self):
         return self.rattrs()
 
     # lookup by name
-    def namelookup(self, did, name, *,
+    def namelookupleaf(self, did, name, *,
+            path=None,
             depth=None):
         rbyd = self.rbyd
         bid = 0
         depth_ = 1
+        path_ = []
 
         while True:
-            found_, rid_, name_ = rbyd.namelookup(did, name)
+            # corrupt branch?
+            if not rbyd:
+                return (bid+(rbyd.weight-1), rbyd, rbyd.weight-1, None,
+                        *((path_,) if path else ()))
+
+            rid_, name_ = rbyd.namelookup(did, name)
+
+            # keep track of path
+            if path:
+                path_.append((bid + rid_, rbyd, rid_, name_))
 
             # find branch tag if there is one
             branch_ = rbyd.lookup(rid_, TAG_BRANCH, 0x3)
@@ -1210,21 +1422,27 @@ class Btree:
             # found another branch
             if branch_ is not None and (
                     not depth or depth_ < depth):
+                block, trunk, cksum = frombranch(branch_.data)
+                rbyd = Rbyd.fetchck(self.bd, block, trunk, name_.weight,
+                        cksum)
+
                 # update our bid
                 bid += rid_ - (name_.weight-1)
-
-                block, trunk, cksum = frombranch(branch_.data)
-                rbyd = Rbyd.fetch(self.bd, block, trunk, cksum)
-                # keep track of expected trunk/weight if corrupted
-                if not rbyd:
-                    rbyd.trunk = trunk
-                    rbyd.weight = name_.weight
-
                 depth_ += 1
 
             # found best match
             else:
-                return found_, bid + rid_, rbyd, rid_, name_
+                return (bid + rid_, rbyd, rid_, name_,
+                        *((path_,) if path else ()))
+
+    def namelookup(self, bid, *,
+            path=None,
+            depth=None):
+        # just discard the rbyd info
+        bid, rbyd, rid, name, *path_ = self.namelookupleaf(did, name,
+                path=path,
+                depth=depth)
+        return bid, name, *path_
 
     # create an rbyd tree for debugging
     def _tree_rtree(self, *,
@@ -1315,22 +1533,10 @@ class Btree:
                             roots[bids[i]] = t
 
                 # remap branches to leaf-roots
-                tree_ = set()
+                tree = {t.map(
-                for t in tree:
+                            lambda x: x[1] == d and x[0] in roots,
-                    if t.a[1] == d and t.a[0] in roots:
+                            lambda x: roots[x[0]].a)
-                        t = TreeBranch(
+                        for t in tree}
-                                roots[t.a[0]].a,
-                                t.b,
-                                t.depth,
-                                t.color)
-                    if t.b[1] == d and t.b[0] in roots:
-                        t = TreeBranch(
-                                t.a,
-                                roots[t.b[0]].a,
-                                t.depth,
-                                t.color)
-                    tree_.add(t)
-                tree = tree_
 
         return tree
 
@@ -1343,7 +1549,7 @@ class Btree:
         tree = set()
         root = None
         branches = {}
-        for bid, rbyd, rid, name, path in self.bids(
+        for bid, name, path in self.bids(
                 path=True,
                 depth=depth):
             # create branch for each jump in path
@@ -1463,7 +1669,7 @@ def main(disk, roots=None, *,
                             if rattr.weight > 1
                             else bid if rattr.weight > 0
                             else '',
-                        21+w_width, rattr,
+                        21+w_width, rattr.repr(),
                         next(xxd(rattr.data, 8), '')
                             if not args.get('raw')
                                 and not args.get('no_truncate')
@@ -1472,8 +1678,7 @@ def main(disk, roots=None, *,
 
                 # show on-disk encoding of tags/data
                 if args.get('raw'):
-                    for o, line in enumerate(xxd(
+                    for o, line in enumerate(xxd(rattr.tdata)):
-                            rbyd.data[rattr.toff:rattr.off])):
                         print('%9s: %*s%*s %s' % (
                                 '%04x' % (rattr.toff + o*16),
                                 t_width, '',
@@ -1553,7 +1758,7 @@ if __name__ == "__main__":
             default='auto',
             help="When to use terminal colors. Defaults to 'auto'.")
     parser.add_argument(
-            '-r', '--raw',
+            '-x', '--raw',
             action='store_true',
             help="Show the raw data including tag encodings.")
     parser.add_argument(
@@ -1581,7 +1786,7 @@ if __name__ == "__main__":
             nargs='?',
             type=lambda x: int(x, 0),
             const=0,
-            help="Depth of tree to show.")
+            help="Depth of the btree to show.")
     parser.add_argument(
             '-e', '--error-on-corrupt',
             action='store_true',

scripts/dbgrbyd.py

@@ -6,6 +6,7 @@ if __name__ == "__main__":
 
 import bisect
 import collections as co
+import functools as ft
 import itertools as it
 import math as mt
 import os
@@ -168,12 +169,17 @@ def xxd(data, width=16):
                     b if b >= ' ' and b <= '~' else '.'
                         for b in map(chr, data[i:i+width])))
 
-def tagrepr(tag, weight=None, size=None, off=None):
+# human readable tag repr
+def tagrepr(tag, weight=None, size=None, *,
+        global_=False,
+        toff=None):
+    # null tags
     if (tag & 0x6fff) == TAG_NULL:
         return '%snull%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
                 ' w%d' % weight if weight else '',
                 ' %d' % size if size else '')
+    # config tags
     elif (tag & 0x6f00) == TAG_CONFIG:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -188,13 +194,23 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'config 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # global-state delta tags
     elif (tag & 0x6f00) == TAG_GDELTA:
-        return '%s%s%s%s' % (
+        if global_:
-                'shrub' if tag & TAG_SHRUB else '',
+            return '%s%s%s%s' % (
-                'grmdelta' if (tag & 0xfff) == TAG_GRMDELTA
+                    'shrub' if tag & TAG_SHRUB else '',
-                    else 'gdelta 0x%02x' % (tag & 0xff),
+                    'grm' if (tag & 0xfff) == TAG_GRMDELTA
-                ' w%d' % weight if weight else '',
+                        else 'gstate 0x%02x' % (tag & 0xff),
-                ' %s' % size if size is not None else '')
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+        else:
+            return '%s%s%s%s' % (
+                    'shrub' if tag & TAG_SHRUB else '',
+                    'grmdelta' if (tag & 0xfff) == TAG_GRMDELTA
+                        else 'gdelta 0x%02x' % (tag & 0xff),
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+    # name tags, includes file types
     elif (tag & 0x6f00) == TAG_NAME:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -206,6 +222,7 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'name 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # structure tags
     elif (tag & 0x6f00) == TAG_STRUCT:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -221,6 +238,7 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'struct 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # custom attributes
     elif (tag & 0x6e00) == TAG_ATTR:
         return '%s%sattr 0x%02x%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -228,37 +246,49 @@ def tagrepr(tag, weight=None, size=None, off=None):
                 ((tag & 0x100) >> 1) ^ (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # alt pointers
     elif tag & TAG_ALT:
         return 'alt%s%s 0x%03x%s%s' % (
                 'r' if tag & TAG_R else 'b',
                 'gt' if tag & TAG_GT else 'le',
                 tag & 0x0fff,
                 ' w%d' % weight if weight is not None else '',
-                ' 0x%x' % (0xffffffff & (off-size))
+                ' 0x%x' % (0xffffffff & (toff-size))
-                    if size and off is not None
+                    if size and toff is not None
                     else ' -%d' % size if size
                     else '')
+    # checksum tags
     elif (tag & 0x7f00) == TAG_CKSUM:
         return 'cksum%s%s%s%s' % (
                 'p' if not tag & 0xfe and tag & TAG_P else '',
                 ' 0x%02x' % (tag & 0xff) if tag & 0xfe else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # note tags
     elif (tag & 0x7f00) == TAG_NOTE:
         return 'note%s%s%s' % (
                 ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # erased-state checksum tags
     elif (tag & 0x7f00) == TAG_ECKSUM:
         return 'ecksum%s%s%s' % (
                 ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # global-checksum delta tags
     elif (tag & 0x7f00) == TAG_GCKSUMDELTA:
-        return 'gcksumdelta%s%s%s' % (
+        if global_:
-                ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
+            return 'gcksum%s%s%s' % (
-                ' w%d' % weight if weight else '',
+                    ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
-                ' %s' % size if size is not None else '')
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+        else:
+            return 'gcksumdelta%s%s%s' % (
+                    ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+    # unknown tags
     else:
         return '0x%04x%s%s' % (
                 tag,
@@ -280,6 +310,54 @@ class TreeBranch(co.namedtuple('TreeBranch', ['a', 'b', 'depth', 'color'])):
                 self.depth,
                 self.color)
 
+    # don't include color in branch comparisons, or else our tree
+    # renderings can end up with inconsistent colors between runs
+    def __eq__(self, other):
+        return (self.a, self.b, self.depth) == (other.a, other.b, other.depth)
+
+    def __ne__(self, other):
+        return (self.a, self.b, self.depth) != (other.a, other.b, other.depth)
+
+    def __hash__(self):
+        return hash((self.a, self.b, self.depth))
+
+    # also order by depth first, which can be useful for reproducibly
+    # prioritizing branches when simplifying trees
+    def __lt__(self, other):
+        return (self.depth, self.a, self.b) < (other.depth, other.a, other.b)
+
+    def __le__(self, other):
+        return (self.depth, self.a, self.b) <= (other.depth, other.a, other.b)
+
+    def __gt__(self, other):
+        return (self.depth, self.a, self.b) > (other.depth, other.a, other.b)
+
+    def __ge__(self, other):
+        return (self.depth, self.a, self.b) >= (other.depth, other.a, other.b)
+
+    # apply a function to a/b while trying to avoid copies
+    def map(self, filter_, map_=None):
+        if map_ is None:
+            filter_, map_ = None, filter_
+
+        a = self.a
+        if filter_ is None or filter_(a):
+            a = map_(a)
+
+        b = self.b
+        if filter_ is None or filter_(b):
+            b = map_(b)
+
+        if a != self.a or b != self.b:
+            return self.__class__(
+                    a if a != self.a else self.a,
+                    b if b != self.b else self.b,
+                    self.depth,
+                    self.color)
+        else:
+            return self
+
+# render some nice ascii trees
 def treerepr(tree, x, depth=None, color=False):
     # find the max depth from the tree
     if depth is None:
@@ -339,17 +417,17 @@ class Bd:
         self.block_count = block_count
 
     def __repr__(self):
-        return '<%s %sx%s>' % (
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
-                self.__class__.__name__,
+
-                self.block_size,
+    def repr(self):
-                self.block_count)
+        return 'bd %sx%s' % (self.block_size, self.block_count)
 
     def read(self, size=-1):
         return self.f.read(size)
 
-    def seek(self, block, off, whence=0):
+    def seek(self, block, off=0, whence=0):
         pos = self.f.seek(block*self.block_size + off, whence)
-        return pos // block_size, pos % block_size
+        return pos // self.block_size, pos % self.block_size
 
     def readblock(self, block):
         self.f.seek(block*self.block_size)
@@ -357,22 +435,40 @@ class Bd:
 
 # tagged data in an rbyd
 class Rattr:
-    def __init__(self, tag, weight, block, toff, off, data):
+    def __init__(self, tag, weight, blocks, toff, tdata, data):
         self.tag = tag
         self.weight = weight
-        self.block = block
+        if isinstance(blocks, int):
+            self.blocks = [blocks]
+        else:
+            self.blocks = list(blocks)
         self.toff = toff
-        self.off = off
+        self.tdata = tdata
         self.data = data
 
+    @property
+    def block(self):
+        return self.blocks[0]
+
+    @property
+    def tsize(self):
+        return len(self.tdata)
+
+    @property
+    def off(self):
+        return self.toff + len(self.tdata)
+
     @property
     def size(self):
         return len(self.data)
 
-    def __repr__(self):
+    def __bytes__(self):
-        return '<%s %s>' % (self.__class__.__name__, self)
+        return self.data
 
-    def __str__(self):
+    def __repr__(self):
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
+
+    def repr(self):
         return tagrepr(self.tag, self.weight, self.size)
 
     def __iter__(self):
@@ -388,14 +484,42 @@ class Rattr:
     def __hash__(self):
         return hash((self.tag, self.weight, self.data))
 
+    # convenience for did/name access
+    def _parse_name(self):
+        # note we return a null name for non-name tags, this is so
+        # vestigial names in btree nodes act as a catch-all
+        if (self.tag & 0xff00) != TAG_NAME:
+            did = 0
+            name = b''
+        else:
+            did, d = fromleb128(self.data)
+            name = self.data[d:]
+
+        # cache both
+        self.did = did
+        self.name = name
+
+    @ft.cached_property
+    def did(self):
+        self._parse_name()
+        return self.did
+
+    @ft.cached_property
+    def name(self):
+        self._parse_name()
+        return self.name
+
 class Ralt:
-    def __init__(self, tag, weight, block, toff, off, jump,
+    def __init__(self, tag, weight, blocks, toff, tdata, jump,
             color=None, followed=None):
         self.tag = tag
         self.weight = weight
-        self.block = block
+        if isinstance(blocks, int):
+            self.blocks = [blocks]
+        else:
+            self.blocks = list(blocks)
         self.toff = toff
-        self.off = off
+        self.tdata = tdata
         self.jump = jump
 
         if color is not None:
@@ -404,15 +528,27 @@ class Ralt:
             self.color = 'r' if tag & TAG_R else 'b'
         self.followed = followed
 
+    @property
+    def block(self):
+        return self.blocks[0]
+
+    @property
+    def tsize(self):
+        return len(self.tdata)
+
+    @property
+    def off(self):
+        return self.toff + len(self.tdata)
+
     @property
     def joff(self):
         return self.toff - self.jump
 
     def __repr__(self):
-        return '<%s %s>' % (self.__class__.__name__, self)
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
 
-    def __str__(self):
+    def repr(self):
-        return tagrepr(self.tag, self.weight, self.jump, self.toff)
+        return tagrepr(self.tag, self.weight, self.jump, toff=self.toff)
 
     def __iter__(self):
         return iter((self.tag, self.weight, self.jump))
@@ -430,19 +566,20 @@ class Ralt:
 
 # our core rbyd type
 class Rbyd:
-    def __init__(self, data, blocks, trunk, weight, rev, eoff, cksum, *,
+    def __init__(self, blocks, trunk, weight, rev, eoff, cksum, data, *,
             gcksumdelta=None,
             corrupt=False):
         if isinstance(blocks, int):
-            blocks = [blocks]
+            self.blocks = [blocks]
-
+        else:
-        self.data = data
+            self.blocks = list(blocks)
-        self.blocks = list(blocks)
         self.trunk = trunk
         self.weight = weight
         self.rev = rev
         self.eoff = eoff
         self.cksum = cksum
+        self.data = data
+
         self.gcksumdelta = gcksumdelta
         self.corrupt = corrupt
 
@@ -459,10 +596,10 @@ class Rbyd:
                     self.trunk)
 
     def __repr__(self):
-        return '<%s %s w%s>' % (
+        return '<%s %s>' % (self.__class__.__name__, self.repr())
-                self.__class__.__name__,
+
-                self.addr(),
+    def repr(self):
-                self.weight)
+        return 'rbyd %s w%s' % (self.addr(), self.weight)
 
     def __bool__(self):
         return not self.corrupt
@@ -478,11 +615,11 @@ class Rbyd:
         return hash((frozenset(self.blocks), self.trunk))
 
     @classmethod
-    def fetch(cls, bd, blocks, trunk=None, cksum=None):
+    def fetch(cls, bd, blocks, trunk=None):
         # multiple blocks? unfortunately this must be a list
         if isinstance(blocks, list):
             # fetch all blocks
-            rbyds = [cls.fetch(bd, block, trunk, cksum) for block in blocks]
+            rbyds = [cls.fetch(bd, block, trunk) for block in blocks]
             # determine most recent revision
             i = 0
             for i_, rbyd in enumerate(rbyds):
@@ -498,6 +635,9 @@ class Rbyd:
             rbyd.blocks += tuple(
                     rbyds[(i+1+j) % len(rbyds)].block
                         for j in range(len(rbyds)-1))
+            # and patch the gcksumdelta if we have one
+            if rbyd.gcksumdelta is not None:
+                rbyd.gcksumdelta.blocks = rbyd.blocks
             return rbyd
 
         block = blocks
@@ -510,9 +650,9 @@ class Rbyd:
                     else block[1] if isinstance(block, tuple)
                     else None)
 
-        # bd can be either a bd reference or preread data
+        # bd can be either a bd reference or a preread block
         #
-        # preread data can be useful for avoiding race conditions
+        # preread blocks can be useful for avoiding race conditions
         # with cksums and shrubs
         if isinstance(bd, Bd):
             # seek/read the block
@@ -522,9 +662,9 @@ class Rbyd:
 
         # fetch the rbyd
         rev = fromle32(data[0:4])
-        cksum_ = 0
+        cksum = 0
-        cksum__ = crc32c(data[0:4])
+        cksum_ = crc32c(data[0:4])
-        cksum___ = cksum__
+        cksum__ = cksum_
         perturb = False
         eoff = 0
         eoff_ = None
@@ -540,10 +680,10 @@ class Rbyd:
         while j_ < len(data) and (not trunk or eoff <= trunk):
             # read next tag
             v, tag, w, size, d = fromtag(data[j_:])
-            if v != parity(cksum___):
+            if v != parity(cksum__):
                 break
-            cksum___ ^= 0x00000080 if v else 0
+            cksum__ ^= 0x00000080 if v else 0
-            cksum___ = crc32c(data[j_:j_+d], cksum___)
+            cksum__ = crc32c(data[j_:j_+d], cksum__)
             j_ += d
             if not tag & TAG_ALT and j_ + size > len(data):
                 break
@@ -551,22 +691,23 @@ class Rbyd:
             # take care of cksums
             if not tag & TAG_ALT:
                 if (tag & 0xff00) != TAG_CKSUM:
-                    cksum___ = crc32c(data[j_:j_+size], cksum___)
+                    cksum__ = crc32c(data[j_:j_+size], cksum__)
 
                     # found a gcksumdelta?
                     if (tag & 0xff00) == TAG_GCKSUMDELTA:
-                        gcksumdelta_ = Rattr(tag, w,
+                        gcksumdelta_ = Rattr(tag, w, block, j_-d,
-                                block, j_-d, d, data[j_:j_+size])
+                                data[j_-d:j_],
+                                data[j_:j_+size])
 
                 # found a cksum?
                 else:
                     # check cksum
-                    cksum____ = fromle32(data[j_:j_+4])
+                    cksum___ = fromle32(data[j_:j_+4])
-                    if cksum___ != cksum____:
+                    if cksum__ != cksum___:
                         break
                     # commit what we have
                     eoff = eoff_ if eoff_ else j_ + size
-                    cksum_ = cksum__
+                    cksum = cksum_
                     trunk_ = trunk__
                     weight = weight_
                     gcksumdelta = gcksumdelta_
@@ -574,7 +715,7 @@ class Rbyd:
                     # update perturb bit
                     perturb = tag & TAG_P
                     # revert to data cksum and perturb
-                    cksum___ = cksum__ ^ (0xfca42daf if perturb else 0)
+                    cksum__ = cksum_ ^ (0xfca42daf if perturb else 0)
 
             # evaluate trunks
             if (tag & 0xf000) != TAG_CKSUM:
@@ -598,7 +739,7 @@ class Rbyd:
                             if trunk and j_ + size > trunk:
                                 eoff_ = j_ + size
                                 eoff = eoff_
-                                cksum_ = cksum___ ^ (
+                                cksum = cksum__ ^ (
                                         0xfca42daf if perturb else 0)
                                 trunk_ = trunk__
                                 weight = weight_
@@ -606,23 +747,34 @@ class Rbyd:
                         trunk___ = 0
 
                 # update canonical checksum, xoring out any perturb state
-                cksum__ = cksum___ ^ (0xfca42daf if perturb else 0)
+                cksum_ = cksum__ ^ (0xfca42daf if perturb else 0)
 
             if not tag & TAG_ALT:
                 j_ += size
 
-        # cksum mismatch?
+        return cls(block, trunk_, weight, rev, eoff, cksum, data,
-        if cksum is not None and cksum_ != cksum:
-            return cls(data, block, 0, 0, rev, 0, cksum_,
-                    corrupt=True)
-
-        return cls(data, block, trunk_, weight, rev, eoff, cksum_,
                 gcksumdelta=gcksumdelta,
                 corrupt=not trunk_)
 
+    @classmethod
+    def fetchck(cls, bd, blocks, trunk, weight, cksum):
+        # try to fetch the rbyd normally
+        rbyd = cls.fetch(bd, blocks, trunk)
+
+        # cksum mismatch? trunk/weight mismatch?
+        if (rbyd.cksum != cksum
+                or rbyd.trunk != trunk
+                or rbyd.weight != weight):
+            # mark as corrupt and keep track of expected trunk/weight
+            rbyd.corrupt = True
+            rbyd.trunk = trunk
+            rbyd.weight = weight
+
+        return rbyd
+
     def lookupnext(self, rid, tag=None, *,
             path=False):
-        if not self:
+        if not self or rid >= self.weight:
             return None, None, *(([],) if path else ())
 
         tag = max(tag or 0, 0x1)
@@ -658,7 +810,8 @@ class Rbyd:
                             color = 'b'
 
                         path_.append(Ralt(
-                                alt, w, self.block, j+jump, j+jump+d, jump,
+                                alt, w, self.blocks, j+jump,
+                                self.data[j+jump:j+jump+d], jump,
                                 color=color,
                                 followed=True))
 
@@ -680,7 +833,8 @@ class Rbyd:
                             color = 'b'
 
                         path_.append(Ralt(
-                                alt, w, self.block, j-d, j, jump,
+                                alt, w, self.blocks, j-d,
+                                self.data[j-d:j], jump,
                                 color=color,
                                 followed=False))
 
@@ -694,7 +848,8 @@ class Rbyd:
                     return None, None, *(([],) if path else ())
 
                 return (rid_,
-                        Rattr(tag_, w_, self.block, j, j+d,
+                        Rattr(tag_, w_, self.blocks, j,
+                            self.data[j:j+d],
                             self.data[j+d:j+d+jump]),
                         *((path_,) if path else ()))
 
@@ -748,7 +903,7 @@ class Rbyd:
             yield rid, name, *path_
             rid += 1
 
-    def rattrs_(self, rid=None, *,
+    def rattrs_(self, rid=None, tag=None, mask=None, *,
             path=False):
         if rid is None:
             rid, tag = -1, 0
@@ -762,24 +917,31 @@ class Rbyd:
                 yield rid, rattr, *path_
                 tag = rattr.tag
         else:
-            tag = 0
+            if tag is None:
+                tag, mask = 0, 0xffff
+            if mask is None:
+                mask = 0
+
+            tag_ = max((tag & ~mask) - 1, 0)
             while True:
-                rid_, rattr, *path_ = self.lookupnext(rid, tag+0x1,
+                rid_, rattr_, *path_ = self.lookupnext(rid, tag_+0x1,
                         path=path)
                 # found end of tree?
-                if rid_ is None or rid_ != rid:
+                if (rid_ is None
+                        or rid_ != rid
+                        or (rattr_.tag & ~mask) != (tag & ~mask)):
                     break
 
-                yield rattr, *path_
+                yield rattr_, *path_
-                tag = rattr.tag
+                tag_ = rattr_.tag
 
-    def rattrs(self, rid=None, *,
+    def rattrs(self, rid=None, tag=None, mask=None, *,
             path=False):
         if rid is None:
-            yield from self.rattrs_(rid,
+            yield from self.rattrs_(rid, tag, mask,
                     path=path)
         else:
-            for rattr, *path_ in self.rattrs_(rid,
+            for rattr, *path_ in self.rattrs_(rid, tag, mask,
                     path=path):
                 if path:
                     yield rattr, *path_
@@ -792,33 +954,25 @@ class Rbyd:
     # lookup by name
     def namelookup(self, did, name):
         # binary search
-        best = (False, None, None, None, None)
+        best = None, None
         lower = 0
         upper = self.weight
         while lower < upper:
-            rid, rattr = self.lookupnext(lower + (upper-1-lower)//2)
+            rid, name_ = self.lookupnext(
+                lower + (upper-1-lower)//2)
             if rid is None:
                 break
 
-            # treat vestigial names as a catch-all
-            if ((rattr.tag == TAG_NAME and rid-(rattr.weight-1) == 0)
-                    or (rattr.tag & 0xff00) != TAG_NAME):
-                did_ = 0
-                name_ = b''
-            else:
-                did_, d = fromleb128(rattr.data)
-                name_ = rattr.data[d:]
-
             # bisect search space
-            if (did_, name_) > (did, name):
+            if (name_.did, name_.name) > (did, name):
-                upper = rid-(w-1)
+                upper = rid-(name_.weight-1)
-            elif (did_, name_) < (did, name):
+            elif (name_.did, name_.name) < (did, name):
                 lower = rid + 1
                 # keep track of best match
-                best = (False, rid, rattr)
+                best = rid, name_
             else:
                 # found a match
-                return True, rid, rattr
+                return rid, name_
 
         return best
 
@@ -932,6 +1086,7 @@ class Rbyd:
             return self._tree_rtree(**args)
 
 
+# show the rbyd log
 def dbg_log(rbyd, *,
         block_size,
         color=False,
@@ -1266,7 +1421,7 @@ def dbg_log(rbyd, *,
                     else '%d-%d' % (rid-(w-1), rid) if w > 1
                     else rid,
                 56+w_width, '%-*s  %s' % (
-                    21+w_width, tagrepr(tag, w, size, j),
+                    21+w_width, tagrepr(tag, w, size, toff=j),
                     next(xxd(data[j+d:j+d+min(size, 8)], 8), '')
                         if not args.get('raw')
                             and not args.get('no_truncate')
@@ -1299,7 +1454,7 @@ def dbg_log(rbyd, *,
                             line,
                             '\x1b[m' if color and j >= rbyd.eoff else ''))
 
-
+# show the rbyd tree
 def dbg_tree(rbyd, *,
         block_size,
         color=False,
@@ -1307,8 +1462,6 @@ def dbg_tree(rbyd, *,
     if not rbyd:
         return
 
-    data = rbyd.data
-
     # precompute tree renderings
     t_width = 0
     if (args.get('tree')
@@ -1337,7 +1490,7 @@ def dbg_tree(rbyd, *,
                     if rattr.weight > 1
                     else rid if rattr.weight > 0 or i == 0
                     else '',
-                21+w_width, rattr,
+                21+w_width, rattr.repr(),
                 next(xxd(rattr.data[:8], 8), '')
                     if not args.get('raw')
                         and not args.get('no_truncate')
@@ -1346,7 +1499,7 @@ def dbg_tree(rbyd, *,
 
         # show on-disk encoding of tags
         if args.get('raw'):
-            for o, line in enumerate(xxd(data[rattr.toff:rattr.off])):
+            for o, line in enumerate(xxd(rattr.tdata)):
                 print('%8s: %*s%*s %s' % (
                         '%04x' % (rattr.toff + o*16),
                         t_width, '',
@@ -1460,7 +1613,7 @@ if __name__ == "__main__":
             action='store_true',
             help="Show the raw tags as they appear in the log.")
     parser.add_argument(
-            '-r', '--raw',
+            '-x', '--raw',
             action='store_true',
             help="Show the raw data including tag encodings.")
     parser.add_argument(

scripts/dbgtag.py

@@ -62,12 +62,17 @@ def fromleb128(data):
             return word, i+1
     return word, len(data)
 
-def tagrepr(tag, weight=None, size=None, off=None):
+# human readable tag repr
+def tagrepr(tag, weight=None, size=None, *,
+        global_=False,
+        toff=None):
+    # null tags
     if (tag & 0x6fff) == TAG_NULL:
         return '%snull%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
                 ' w%d' % weight if weight else '',
                 ' %d' % size if size else '')
+    # config tags
     elif (tag & 0x6f00) == TAG_CONFIG:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -82,13 +87,23 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'config 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # global-state delta tags
     elif (tag & 0x6f00) == TAG_GDELTA:
-        return '%s%s%s%s' % (
+        if global_:
-                'shrub' if tag & TAG_SHRUB else '',
+            return '%s%s%s%s' % (
-                'grmdelta' if (tag & 0xfff) == TAG_GRMDELTA
+                    'shrub' if tag & TAG_SHRUB else '',
-                    else 'gdelta 0x%02x' % (tag & 0xff),
+                    'grm' if (tag & 0xfff) == TAG_GRMDELTA
-                ' w%d' % weight if weight else '',
+                        else 'gstate 0x%02x' % (tag & 0xff),
-                ' %s' % size if size is not None else '')
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+        else:
+            return '%s%s%s%s' % (
+                    'shrub' if tag & TAG_SHRUB else '',
+                    'grmdelta' if (tag & 0xfff) == TAG_GRMDELTA
+                        else 'gdelta 0x%02x' % (tag & 0xff),
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+    # name tags, includes file types
     elif (tag & 0x6f00) == TAG_NAME:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -100,6 +115,7 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'name 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # structure tags
     elif (tag & 0x6f00) == TAG_STRUCT:
         return '%s%s%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -115,6 +131,7 @@ def tagrepr(tag, weight=None, size=None, off=None):
                     else 'struct 0x%02x' % (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # custom attributes
     elif (tag & 0x6e00) == TAG_ATTR:
         return '%s%sattr 0x%02x%s%s' % (
                 'shrub' if tag & TAG_SHRUB else '',
@@ -122,37 +139,49 @@ def tagrepr(tag, weight=None, size=None, off=None):
                 ((tag & 0x100) >> 1) ^ (tag & 0xff),
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # alt pointers
     elif tag & TAG_ALT:
         return 'alt%s%s 0x%03x%s%s' % (
                 'r' if tag & TAG_R else 'b',
                 'gt' if tag & TAG_GT else 'le',
                 tag & 0x0fff,
                 ' w%d' % weight if weight is not None else '',
-                ' 0x%x' % (0xffffffff & (off-size))
+                ' 0x%x' % (0xffffffff & (toff-size))
-                    if size and off is not None
+                    if size and toff is not None
                     else ' -%d' % size if size
                     else '')
+    # checksum tags
     elif (tag & 0x7f00) == TAG_CKSUM:
         return 'cksum%s%s%s%s' % (
                 'p' if not tag & 0xfe and tag & TAG_P else '',
                 ' 0x%02x' % (tag & 0xff) if tag & 0xfe else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # note tags
     elif (tag & 0x7f00) == TAG_NOTE:
         return 'note%s%s%s' % (
                 ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # erased-state checksum tags
     elif (tag & 0x7f00) == TAG_ECKSUM:
         return 'ecksum%s%s%s' % (
                 ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
                 ' w%d' % weight if weight else '',
                 ' %s' % size if size is not None else '')
+    # global-checksum delta tags
     elif (tag & 0x7f00) == TAG_GCKSUMDELTA:
-        return 'gcksumdelta%s%s%s' % (
+        if global_:
-                ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
+            return 'gcksum%s%s%s' % (
-                ' w%d' % weight if weight else '',
+                    ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
-                ' %s' % size if size is not None else '')
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+        else:
+            return 'gcksumdelta%s%s%s' % (
+                    ' 0x%02x' % (tag & 0xff) if tag & 0xff else '',
+                    ' w%d' % weight if weight else '',
+                    ' %s' % size if size is not None else '')
+    # unknown tags
     else:
         return '0x%04x%s%s' % (
                 tag,