scripts: Fixed O(n^2) slicing in Rbyd.fetch

Do you see the O(n^2) behavior in this loop?

  j = 0
  while j < len(data):
      word, d = fromleb(data[j:])
      j += d

The slice, data[j:], creates a O(n) copy every iteration of the loop.

A bit tricky. Or at least I found it tricky to notice. Maybe because
array indexing being cheap is baked into my brain...

Long story short, this repeated slicing resulted in O(n^2) behavior in
Rbyd.fetch and probably some other functions. Even though we don't care
_too_ much about performance in these scripts, having Rbyd.fetch run in
O(n^2) isn't great.

Tweaking all from* functions to take an optional index solves this, at
least on paper.

---

In practice I didn't actually find any measurable performance gain. I
guess array slicing in Python is optimized enough that the constant
factor takes over?

(Maybe it's being helped by us limiting Rbyd.fetch to block_size in most
scripts? I haven't tested NAND block sizes yet...)

Still, it's good to at least know this isn't a bottleneck.
This commit is contained in:
Christopher Haster
2025-04-14 14:27:44 -05:00
parent 8b11cea3f2
commit 0cea8b96fb
9 changed files with 317 additions and 293 deletions
+14 -11
View File
@@ -64,21 +64,24 @@ def openio(path, mode='r', buffering=-1):
else:
return open(path, mode, buffering)
def fromleb128(data):
def fromleb128(data, j=0):
word = 0
for i, b in enumerate(data):
word |= ((b & 0x7f) << 7*i)
d = 0
while j+d < len(data):
b = data[j+d]
word |= (b & 0x7f) << 7*d
word &= 0xffffffff
if not b & 0x80:
return word, i+1
return word, d+1
d += 1
return word, len(data)
def fromtag(data):
data = data.ljust(4, b'\0')
tag = struct.unpack('>H', data[:2])[0]
weight, d = fromleb128(data[2:])
size, d_ = fromleb128(data[2+d:])
return tag>>15, tag&0x7fff, weight, size, 2+d+d_
def fromtag(data, j=0):
d = 0
tag = struct.unpack('>H', data[j:j+2].ljust(2, b'\0'))[0]; d += 2
weight, d_ = fromleb128(data, j+d); d += d_
size, d_ = fromleb128(data, j+d); d += d_
return tag>>15, tag&0x7fff, weight, size, d
# human readable tag repr
def tagrepr(tag, weight=None, size=None, *,
@@ -244,7 +247,7 @@ def dbg_tags(data):
else:
j = 0
while j < len(data):
v, tag, w, size, d = fromtag(data[j:])
v, tag, w, size, d = fromtag(data, j)
lines.append((
' '.join('%02x' % b for b in data[j:j+d]),
tagrepr(tag, w, size)))