scripts: csv.py: Optional by fields for unique enumerates/accumulates

This extends csv.py's enumerate/accumulate exprs with optional by field
arguments. Each set of by fields gets its own state, allowing multiple
parallel enumerates/accumulates to be processed simultaneously.

This is especially useful when the number of by field sets is unknown.
In theory you could split/merge each by field set with a separate csv.py
call, but it'd be a real pain.

Consider some bench results:

  case,n,simtime
  bench_rbyd,1,100
  bench_rbyd,2,10
  bench_rbyd,3,100
  bench_btree,1,200
  bench_rbyd,4,10
  bench_btree,2,20
  bench_btree,3,2000
  bench_btree,4,200

It was a bit awkward to handle these with csv.py's accumulate, as
accumulate operated strictly per-row, ignoring the case field.

But now with optional by fields:

  $ ./scripts/csv.py test.csv \
        -bcase -bn \
        -fsimtime='accumulate(simtime, case)'
  case,n           simtime
  bench_btree,1        200
  bench_btree,2        220
  bench_btree,3       2220
  bench_btree,4       2420
  bench_rbyd,1         100
  bench_rbyd,2         110
  bench_rbyd,3         210
  bench_rbyd,4         220
  TOTAL               5700

Note that these by fields are a bit special in csv.py's grammar. So far,
they are the only fields in field exprs that aren't typechecked. The
alternative would be string types in csv.py, but I'm not sure I want to
go that far.

---

It's tempting to try to invert this logic (accumulate(simtime, n)), but
I'm not sure how it would work internally. The duplicate by fields
("case") do get annoying, but specifying them in the expr helps make the
relevant state explicit.

Keep in mind we don't evaluate the actual by fields until much later in
csv.py. Entangling these stages risks confusion (-ba='%(b)s'
-c='enumerate(n)'? hidden by fields? overlapping by+field fields?).

scripts/csv.py

@@ -789,47 +789,83 @@ class CsvExpr:
                 return CsvGStddev()([v.eval(fields, state) for v in self])
 
     # enumerate exprs
-    @func('enumerate', '')
+    @func('enumerate', '[*by]')
     class Enumerate(Expr):
-        """A number incremented each result"""
+        """A [per by] number incremented for each result"""
         def fields(self):
+            # don't typecheck by fields
             return set()
 
         def type(self, types={}):
+            # don't typecheck, but make sure we can read by fields
+            for v in self:
+                if not isinstance(v, CsvExpr.Field):
+                    raise CsvExpr.Error("complicated by field? %s" % v)
             return CsvInt
 
         def fold(self, types={}):
+            # don't typecheck by fields
             return CsvSum, None
 
         def eval(self, fields={}, state=None):
             if state is None:
                 return CsvInt(0)
-            # enumerate
-            v = state.get(('enumerate', id(self)))
-            if v is None:
-                v = 0
-            else:
-                v += 1
-            # keep track of unique enumerate state
-            state[('enumerate', id(self))] = v
-            return CsvInt(v)
 
-    @func('accumulate', 'a')
+            # enumerate
-    class Accumulate(Expr):
+            k = ['enumerate', id(self)]
-        """A running sum across results"""
+            for v in self:
-        def eval(self, fields={}, state=None):
+                if v.a not in fields:
-            v = self.a.eval(fields, state)
+                    raise CsvExpr.Error("unknown field? %s" % v.a)
-            if state is None:
+                k.append(fields[v.a])
-                return v
+            k = tuple(k)
-            # accumulate
+            x = state.get(k)
-            v_ = state.get(('accumulate', id(self)))
+            if x is None:
-            if v_ is None:
+                x = 0
-                v_ = v
             else:
-                v_ += v
+                x += 1
+            # keep track of unique enumerate state
+            state[k] = x
+            return CsvInt(x)
+
+    @func('accumulate', 'a[, *by]')
+    class Accumulate(Expr):
+        """A [per by] running sum across results"""
+        def fields(self):
+            # don't typecheck by fields
+            return self.a.fields()
+
+        def type(self, types={}):
+            # don't typecheck, but make sure we can read by fields
+            t = self.a.type(types)
+            for v in it.islice(self, 1, None):
+                if not isinstance(v, CsvExpr.Field):
+                    raise CsvExpr.Error("complicated by field? %s" % v)
+            return t
+
+        def fold(self, types={}):
+            # don't typecheck by fields
+            return self.a.fold(types)
+
+        def eval(self, fields={}, state=None):
+            y = self.a.eval(fields, state)
+            if state is None:
+                return y
+
+            # accumulate
+            k = ['accumulate', id(self)]
+            for v in it.islice(self, 1, None):
+                if v.a not in fields:
+                    raise CsvExpr.Error("unknown field? %s" % v.a)
+                k.append(fields[v.a])
+            k = tuple(k)
+            x = state.get(k)
+            if x is None:
+                x = y
+            else:
+                x += y
             # keep track of unique accumulate state
-            state[('accumulate', id(self))] = v_
+            state[k] = x
-            return v_
+            return x
 
     # functions
     @func('ratio', 'a')