Implemented scratch file basics

"Scratch files" are a new file type added to solve the zero-sized
file problem. Though they have a few other uses that may be quite
valuable.

The "zero-sized file problem" is a common surprise for users, where what
seems like a simple file create+write operation:

  lfs_file_open(&lfs, &file, "hi",
          LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL);
  lfs_file_write(&lfs, &file, "hello!", strlen("hello!"));
  lfs_file_close(&lfs, &file);

Can end up create a zero-sized file under powerloss, breaking user
assumptions and their code.

The tricky thing is that this is actually correct behavior as defined by
POSIX. `open` with O_CREAT creats a file entry immediately, which is
initially zero-sized. And the fact that power can be lost between `open`
and `close` isn't really avoidable.

But this is a common enough footgun that it's probably worth deviating
from POSIX here.

But how to avoid zero-sized files exactly? First thought: Delay the file
creation until sync/close, tracking uncreated files in-device until
then. This solves the problem and avoids any intermediary state if we
lose power, but came with a number of headaches:

1. Since we delay file creation, we don't immediately write the filename
   to disk on open. This implies we need to keep the filename allocated
   in RAM until the first sync/close call.

   The requirement to keep the filename allocated for new files until
   first sync/close could be added to open, and with the option to call
   sync immediately to save the filename (and accept the risk of
   zero-sized files), I don't think it would be _that_ bad of an API.

   But it would still be pretty bad. Extra bad because 1. there's no
   way to warn on misuse at compile-time, 2. use-after-free bugs have a
   tendency to go unnoticed annoyingly often, 3. it's a regression from
   the previous API, and 4. who the heck reads the more-or-less same
   `open` documentation for every filesystem they adopt.

2. Without an allocated mid, tracking files internally gets a lot
   harder. The best option I could think of was to keep the opened-file
   linked-list sorted by mid + (in-device) file name.

   This did not feel like a great solutiona and was going to add more
   code cost.

3. Handling mdir splits containing uncreated files adds another
   headache. Complicated lfsr_mdir_estimate further as it needs to
   decide in which mdir the uncreated files will end up, and potentially
   split on a filename that isn't even created yet.

4. Since the number of uncreated files can be potentially unbounded, you
   can't prevent an mdir from filling up with only uncreated files. On
   disk this ends up looking like an "empty" mdir, which need specially
   handling in littlefs to reclaim after powerloss.

   Support for empty mdirs -- the orphaned mdir scan -- was already
   added earlier. We already scan each mdir to build gstate, so it
   doesn't really add much cost.

Notice that last bullet point? We already scan each mdir during mount.
Why not, instead of scanning for orphaned mdirs, scan for orphaned
files?

So this leads to the idea of "scratch files". Instead of actually
delaying file creation, fake it. Create a scratch file during open, and
on the first sync/close, convert it to a regular file. If we lose power,
scan for scratch files during mount, and remove them on first write.

Some tradeoffs:

1. The orphan scan for scratch files is a bit more expensive than for
   mdirs on storage with large block sizes. We need to look at each file
   entry vs just each mdir, which pushed the runtime up to O(BlogB) vs
   O(B).

   Though if you also consider large mtrees, the worst case is still
   O(nlogn).

2. Creating intermediate scratch files adds another commit to file
   creation.

   This is probably not a big issue for flash, but may be more of a
   concern on devices with large prog sizes.

3. Scratch files complicate unrelated mkdir/rename/etc code a bit, since
   we need to consider what happens when the dest is a scratch file.

But the end result is simple. And simple is good. Both for
implementation headaches, and code size. Even if the on-disk state is
conceptually more complicated.

You may have noticed these scratch files are basically isomorphic to
just setting an "uncreated" flag on the file, and that's true. There may
have been a simpler route to end up with the design, but hey, as long as
it works.

As a plus, scratch files present a solution for a couple other things:

1. Removing an open file can become a scratch file until closed.

2. Scratch files can be used as temporary files. Open a file with
   O_DESYNC and never call sync and you have yourself a temporary file.

   Maybe in the future we should add O_TMPFILE to avoid the need for
   unique filenames, but that is low priority.

lfs.c

@@ -617,6 +617,7 @@ enum lfsr_tag {
     LFSR_TAG_NAME           = 0x0200,
     LFSR_TAG_REG            = 0x0201,
     LFSR_TAG_DIR            = 0x0202,
+    LFSR_TAG_SCRATCH        = 0x0203,
     LFSR_TAG_BOOKMARK       = 0x0204,
 
     // struct tags
@@ -6939,6 +6940,11 @@ static int lfsr_mtree_pathlookup(lfs_t *lfs, const char *path,
             return LFS_ERR_NOENT;
         }
 
+        // pretend scratch files don't exist
+        if (tag == LFSR_TAG_SCRATCH) {
+            return LFS_ERR_NOENT;
+        }
+
         // go on to next name
         name += name_size;
 next:;
@@ -7759,9 +7765,6 @@ static int lfsr_mountinited(lfs_t *lfs) {
 
     // traverse the mtree rooted at mroot 0x{1,0}
     //
-    // note that lfsr_traversal_next will update our mroot/mtree
-    // based on what mroots it finds
-    //
     // we do validate btree inner nodes here, how can we trust our
     // mdirs are valid if we haven't checked the btree inner nodes at
     // least once?
@@ -7796,17 +7799,6 @@ static int lfsr_mountinited(lfs_t *lfs) {
                             *lfsr_mdir_mptr(&tinfo.u.mdir),
                             lfsr_mweight(lfs));
                 }
-
-                // found an empty non-mroot mdir? this should only happen
-                // if we lost power
-                if (tinfo.u.mdir.rbyd.weight == 0) {
-                    LFS_DEBUG("Found orphaned mdir %"PRId32" "
-                            "0x{%"PRIx32",%"PRIx32"}",
-                            tinfo.u.mdir.mid >> lfs->mbits,
-                            tinfo.u.mdir.rbyd.blocks[0],
-                            tinfo.u.mdir.rbyd.blocks[1]);
-                    lfs->hasorphans = true;
-                }
             }
 
             // collect any gdeltas from this mdir
@@ -7815,6 +7807,27 @@ static int lfsr_mountinited(lfs_t *lfs) {
                 return err;
             }
 
+            // check for any scratch files
+            for (lfs_size_t i = 0;
+                    i < (lfs_size_t)tinfo.u.mdir.rbyd.weight;
+                    i++) {
+                err = lfsr_mdir_lookup(lfs, &tinfo.u.mdir,
+                        i, LFSR_TAG_SCRATCH,
+                        NULL);
+                if (err && err != LFS_ERR_NOENT) {
+                    return err;
+                }
+
+                // found a scratch file? these must be orphaned
+                if (err != LFS_ERR_NOENT) {
+                    LFS_DEBUG("Found orphaned scratch file "
+                            "%"PRId32".%"PRId32,
+                            lfsr_mid_bid(lfs, tinfo.u.mdir.mid) >> lfs->mbits,
+                            i);
+                    lfs->hasorphans = true;
+                }
+            }
+
         // found an mtree inner-node?
         } else if (tinfo.tag == LFSR_TAG_BRANCH) {
             // found the root of the mtree? keep track of this
@@ -8157,44 +8170,54 @@ static int lfsr_fs_fixgrm(lfs_t *lfs) {
 }
 
 static int lfsr_fs_fixorphans(lfs_t *lfs) {
-    // traverse the filesystem and drop any orphaned mdirs
+    // traverse the filesystem and remove any orphaned scratch files
     //
     // note this never takes longer than lfsr_mount
     //
-    lfsr_traversal_t traversal = LFSR_TRAVERSAL(0);
+    lfsr_mdir_t mdir;
+    int err = lfsr_mtree_lookup(lfs, 0, &mdir);
+    if (err) {
+        LFS_ASSERT(err != LFS_ERR_NOENT);
+        return err;
+    }
+
     while (true) {
-        lfsr_tinfo_t tinfo;
+        // are we a scratch file?
-        int err = lfsr_traversal_read(lfs, &traversal, &tinfo);
+        err = lfsr_mdir_lookup(lfs, &mdir, mdir.mid, LFSR_TAG_SCRATCH,
+                NULL);
+        if (err && err != LFS_ERR_NOENT) {
+            return err;
+        }
+
+        if (err != LFS_ERR_NOENT) {
+            // remove scratch file
+            err = lfsr_mdir_commit(lfs, &mdir, LFSR_ATTRS(
+                    LFSR_ATTR(mdir.mid, RM, -1, NULL())));
+            if (err) {
+                return err;
+            }
+
+            // seek in case our mdir was dropped
+            err = lfsr_mtree_seek(lfs, &mdir, 0);
             if (err) {
                 if (err == LFS_ERR_NOENT) {
                     break;
                 }
                 return err;
             }
-
+        } else {
-        // found an orphaned mdir? drop
+            // lookup next entry
-        if (tinfo.tag == LFSR_TAG_MDIR
+            err = lfsr_mtree_seek(lfs, &mdir, 1);
-                && tinfo.u.mdir.mid != -1
-                && tinfo.u.mdir.rbyd.weight == 0) {
-            err = lfsr_mdir_drop(lfs, &tinfo.u.mdir);
             if (err) {
+                if (err == LFS_ERR_NOENT) {
+                    break;
+                }
                 return err;
             }
-
-            // TODO should we have a function for this?
-            // TODO should traversals be "opened" and updated by
-            // lfsr_mdir_commit/drop?
-            //
-            // dropping an orphan changes our mtree, we need to partially
-            // invalidate out traversal
-            LFS_ASSERT(traversal.state == LFSR_TRAVERSAL_MDIR);
-            traversal.state = LFSR_TRAVERSAL_MDIR;
-            traversal.u.mtraversal.bid -= lfsr_mweight(lfs);
-            traversal.u.mtraversal.rid = traversal.u.mtraversal.bid;
-            traversal.u.mtraversal.branch = lfs->mtree.u.btree;
         }
     }
 
+    lfs->hasorphans = false;
     return 0;
 }
 
@@ -8218,14 +8241,13 @@ static int lfsr_fs_preparemutation(lfs_t *lfs) {
         lfs_alloc_ckpoint(lfs);
     }
 
-    // fix orphaned mdirs
+    // fix orphaned scratch files
     //
-    // this must happen after fixgrm, since dropping mdirs risks outdating
+    // this must happen after fixgrm, since removing scratch files risks
-    // the grm, fixgrm can also create temporary orphans, but it should
+    // outdating the grm
-    // immediately clean them up
     //
     if (lfs->hasorphans) {
-        LFS_DEBUG("Fixing orphaned mdirs...");
+        LFS_DEBUG("Fixing orphaned scratch files...");
         pl = true;
 
         int err = lfsr_fs_fixorphans(lfs);
@@ -8608,36 +8630,16 @@ int lfsr_rename(lfs_t *lfs, const char *old_path, const char *new_path) {
     return lfsr_fs_fixgrm(lfs);
 }
 
-// common stat function once we have an mdir
+// this just populates the info struct based on what we found
-static int lfsr_mdir_stat(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_mid_t mid,
+static int lfsr_stat_(lfs_t *lfs, const lfsr_mdir_t *mdir,
-        lfsr_sdid_t did, struct lfs_info *info) {
+        lfsr_tag_t tag, lfsr_data_t name,
-    // lookup our name tag
+        struct lfs_info *info) {
-    lfsr_tag_t tag;
-    lfsr_data_t data;
-    int err = lfsr_mdir_lookupwide(lfs, mdir, mid, LFSR_TAG_NAME,
-            &tag, &data);
-    if (err) {
-        return err;
-    }
-
-    // get our did
-    lfsr_did_t did_;
-    err = lfsr_data_readleb128(lfs, &data, (int32_t*)&did_);
-    if (err) {
-        return err;
-    }
-
-    // did mismatch? this terminates dir reads
-    if (did != -1 && did_ != (lfsr_did_t)did) {
-        return LFS_ERR_NOENT;
-    }
-
     // get file type from the tag
     info->type = lfsr_tag_subtype(tag);
 
-    // get file name from the name entry
+    // read the file name
-    LFS_ASSERT(lfsr_data_size(&data) <= LFS_NAME_MAX);
+    LFS_ASSERT(lfsr_data_size(&name) <= LFS_NAME_MAX);
-    lfs_ssize_t name_size = lfsr_data_read(lfs, &data,
+    lfs_ssize_t name_size = lfsr_data_read(lfs, &name,
             info->name, LFS_NAME_MAX);
     if (name_size < 0) {
         return name_size;
@@ -8651,7 +8653,7 @@ static int lfsr_mdir_stat(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_mid_t mid,
         // inlined?
         lfsr_tag_t tag;
         lfsr_data_t data;
-        err = lfsr_mdir_lookupnext(lfs, mdir, mid, LFSR_TAG_DATA,
+        int err = lfsr_mdir_lookupnext(lfs, mdir, mdir->mid, LFSR_TAG_DATA,
                 &tag, &data);
         if (err && err != LFS_ERR_NOENT) {
             return err;
@@ -8697,7 +8699,9 @@ int lfsr_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
     }
 
     // fill out our info struct
-    return lfsr_mdir_stat(lfs, &mdir, mdir.mid, -1, info);
+    return lfsr_stat_(lfs, &mdir,
+            tag, LFSR_DATA_BUF(name, name_size),
+            info);
 }
 
 int lfsr_dir_open(lfs_t *lfs, lfsr_dir_t *dir, const char *path) {
@@ -8779,14 +8783,39 @@ int lfsr_dir_read(lfs_t *lfs, lfsr_dir_t *dir, struct lfs_info *info) {
         return err;
     }
 
-    // fill out our info struct
+    while (true) {
-    //
+        // lookup the next name tag
-    // this will return LFS_ERR_NOENT if our dids mismatch
+        lfsr_tag_t tag;
-    err = lfsr_mdir_stat(lfs, &dir->mdir, dir->mdir.mid, dir->did, info);
+        lfsr_data_t data;
+        err = lfsr_mdir_lookupwide(lfs, &dir->mdir,
+                dir->mdir.mid, LFSR_TAG_NAME,
+                &tag, &data);
         if (err) {
             return err;
         }
 
+        // get the did
+        lfsr_did_t did;
+        err = lfsr_data_readleb128(lfs, &data, (int32_t*)&did);
+        if (err) {
+            return err;
+        }
+
+        // did mismatch? this terminates the dir read
+        if (did != dir->did) {
+            return LFS_ERR_NOENT;
+        }
+
+        // skip scratch files, we pretend these don't exist
+        if (tag != LFSR_TAG_SCRATCH) {
+            // fill out our info struct
+            err = lfsr_stat_(lfs, &dir->mdir, tag, data,
+                    info);
+            if (err) {
+                return err;
+            }
+        }
+
         // eagerly look up the next entry
         err = lfsr_mtree_seek(lfs, &dir->mdir, 1);
         if (err && err != LFS_ERR_NOENT) {
@@ -8794,7 +8823,10 @@ int lfsr_dir_read(lfs_t *lfs, lfsr_dir_t *dir, struct lfs_info *info) {
         }
         dir->pos += 1;
 
+        if (tag != LFSR_TAG_SCRATCH) {
             return 0;
+        }
+    }
 }
 
 int lfsr_dir_seek(lfs_t *lfs, lfsr_dir_t *dir, lfs_soff_t off) {
@@ -8960,6 +8992,10 @@ static inline bool lfsr_f_isunsync(uint32_t flags) {
     return flags & LFS_F_UNSYNC;
 }
 
+static inline bool lfsr_f_isuncreat(uint32_t flags) {
+    return flags & LFS_F_UNCREAT;
+}
+
 static inline lfs_off_t lfsr_file_size_(const lfsr_file_t *file) {
     return lfs_max32(
             file->buffer_pos + file->buffer_size,
@@ -8977,6 +9013,15 @@ int lfsr_file_opencfg(lfs_t *lfs, lfsr_file_t *file,
         const struct lfs_file_config *cfg) {
     // don't allow the forbidden mode!
     LFS_ASSERT((flags & 3) != 3);
+    // these flags require a writable file
+    LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_iscreat(flags));
+    LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_isexcl(flags));
+    LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_istrunc(flags));
+    LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_isappend(flags));
+    // these flags are internal and shouldn't be provided by the user
+    LFS_ASSERT(!lfsr_f_isunflush(flags));
+    LFS_ASSERT(!lfsr_f_isunsync(flags));
+    LFS_ASSERT(!lfsr_f_isuncreat(flags));
 
     if (!lfsr_o_isrdonly(flags)) {
         // prepare our filesystem for writing
@@ -9018,20 +9063,17 @@ int lfsr_file_opencfg(lfs_t *lfs, lfsr_file_t *file,
             return LFS_ERR_NAMETOOLONG;
         }
 
-        // create our entry
+        // create a scratch entry, this reserves the mid until first sync
-        //
-        // note this risks creating a zero-length file if we lose power here,
-        // but it's the only way for us to save the file name.
-        //
-        // TODO or is it? ;)
         err = lfsr_mdir_commit(lfs, &file->mdir, LFSR_ATTRS(
                 LFSR_ATTR(file->mdir.mid,
-                    REG, +1, CAT(
+                    SCRATCH, +1, CAT(
                         LFSR_DATA_LEB128(did),
                         LFSR_DATA_BUF(name, name_size)))));
         if (err) {
             return err;
         }
+        file->flags |= LFS_F_UNSYNC | LFS_F_UNCREAT;
+
     } else {
         if (lfsr_o_isexcl(flags)) {
             // oh, we really wanted to create a new entry
@@ -10765,23 +10807,59 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) {
         lfs_alloc_ckpoint(lfs);
 
         // commit our file's metadata
+        lfsr_attr_t attrs[2];
+        lfs_size_t attr_count = 0;
         uint8_t buf[LFSR_BTREE_DSIZE];
-        err = lfsr_mdir_commit(lfs, &file->mdir, LFSR_ATTRS(
+        lfs_size_t buf_size = 0;
-                (lfsr_f_isunflush(file->flags) && file->buffer_size == 0)
+
-                    ? LFSR_ATTR(file->mdir.mid,
+        // not created yet? need to convert scratch file to normal file
+        if (lfsr_f_isuncreat(file->flags)) {
+            lfsr_data_t data;
+            err = lfsr_mdir_lookup(lfs, &file->mdir,
+                    file->mdir.mid, LFSR_TAG_SCRATCH,
+                    &data);
+            if (err) {
+                // we must have a scratch file at this point
+                LFS_ASSERT(err != LFS_ERR_NOENT);
+                goto failed;
+            }
+
+            attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
+                    WIDE(REG), 0, DATA(data));
+        }
+
+        // commit the file state
+
+        // null? no attr?
+        if (lfsr_f_isunflush(file->flags) && file->buffer_size == 0) {
+            attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
                     WIDE(RM(STRUCT)), 0,
-                        NULL())
+                    NULL());
-                : (lfsr_f_isunflush(file->flags))
+        // small file inlined in mdir?
-                    ? LFSR_ATTR(file->mdir.mid,
+        } else if (lfsr_f_isunflush(file->flags)) {
+            attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
                     WIDE(DATA), 0,
-                        BUF(file->buffer, file->buffer_size))
+                    BUF(file->buffer, file->buffer_size));
-                : (lfsr_ftree_isbshrub(&file->mdir, &file->ftree))
+        // bshrub?
-                    ? LFSR_ATTR(file->mdir.mid,
+        } else if (lfsr_ftree_isbshrub(&file->mdir, &file->ftree)) {
+            attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
                     WIDE(SHRUBTRUNK), 0,
-                        SHRUBTRUNK(&file->ftree_.u.bshrub))
+                    SHRUBTRUNK(&file->ftree_.u.bshrub));
-                    : LFSR_ATTR(file->mdir.mid,
+        // btree?
+        } else if (lfsr_ftree_isbtree(&file->mdir, &file->ftree)) {
+            attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
                     WIDE(BTREE), 0,
-                        FROMBTREE(&file->ftree.u.btree, buf))));
+                    FROMBTREE(&file->ftree.u.btree, &buf[buf_size]));
+            buf_size += LFSR_BTREE_DSIZE;
+        } else {
+            LFS_UNREACHABLE();
+        }
+
+        LFS_ASSERT(attr_count <= sizeof(attrs)/sizeof(lfsr_attr_t));
+        LFS_ASSERT(buf_size <= sizeof(buf));
+
+        err = lfsr_mdir_commit(lfs, &file->mdir,
+                attrs, attr_count);
         if (err) {
             goto failed;
         }
@@ -10796,6 +10874,9 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) {
                 && file_->mdir.mid == file->mdir.mid
                 // don't double update
                 && file_ != file) {
+            // notify all files of creation
+            file_->flags &= ~LFS_F_UNCREAT;
+
             // mark desynced files an unsynced
             if (lfsr_o_isdesync(file_->flags)) {
                 file_->flags |= LFS_F_UNSYNC;
@@ -10818,7 +10899,7 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) {
     }
 
     // mark as synced
-    file->flags &= ~LFS_F_UNSYNC & ~LFS_O_DESYNC;
+    file->flags &= ~LFS_F_UNSYNC & ~LFS_F_UNCREAT & ~LFS_O_DESYNC;
     return 0;
 
 failed:;

lfs.h

@@ -139,6 +139,7 @@ enum lfs_open_flags {
     // internally used flags
     LFS_F_UNFLUSH   = 0x1000,   // File's data does not match storage
     LFS_F_UNSYNC    = 0x2000,   // File's metadata does not match storage
+    LFS_F_UNCREAT   = 0x4000,   // File does not exist yet
 };
 
 // File seek flags

scripts/dbgbmap.py

@@ -26,6 +26,7 @@ TAG_GRMDELTA        = 0x0100
 TAG_NAME            = 0x0200
 TAG_REG             = 0x0201
 TAG_DIR             = 0x0202
+TAG_SCRATCH         = 0x0203
 TAG_BOOKMARK        = 0x0204
 TAG_STRUCT          = 0x0300
 TAG_DATA            = 0x0300

scripts/dbgbtree.py

@@ -24,6 +24,7 @@ TAG_GRMDELTA        = 0x0100
 TAG_NAME            = 0x0200
 TAG_REG             = 0x0201
 TAG_DIR             = 0x0202
+TAG_SCRATCH         = 0x0203
 TAG_BOOKMARK        = 0x0204
 TAG_STRUCT          = 0x0300
 TAG_DATA            = 0x0300
@@ -187,6 +188,7 @@ def tagrepr(tag, w, size, off=None):
             'name' if (tag & 0xfff) == TAG_NAME
                 else 'reg' if (tag & 0xfff) == TAG_REG
                 else 'dir' if (tag & 0xfff) == TAG_DIR
+                else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
                 else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
                 else 'name 0x%02x' % (tag & 0xff),
             ' w%d' % w if w else '',

scripts/dbglfs.py

@@ -25,6 +25,7 @@ TAG_GRMDELTA        = 0x0100
 TAG_NAME            = 0x0200
 TAG_REG             = 0x0201
 TAG_DIR             = 0x0202
+TAG_SCRATCH         = 0x0203
 TAG_BOOKMARK        = 0x0204
 TAG_STRUCT          = 0x0300
 TAG_DATA            = 0x0300
@@ -218,6 +219,7 @@ def tagrepr(tag, w, size, off=None):
             'name' if (tag & 0xfff) == TAG_NAME
                 else 'reg' if (tag & 0xfff) == TAG_REG
                 else 'dir' if (tag & 0xfff) == TAG_DIR
+                else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
                 else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
                 else 'name 0x%02x' % (tag & 0xff),
             ' w%d' % w if w else '',
@@ -1179,7 +1181,7 @@ class GState:
             yield grepr(tag, data), tag, data
 
 def frepr(mdir, rid, tag):
-    if tag == TAG_REG:
+    if tag == TAG_REG or tag == TAG_SCRATCH:
         size = 0
         structs = []
         # inlined data?
@@ -1205,7 +1207,9 @@ def frepr(mdir, rid, tag):
             weight, block, trunk, cksum = frombtree(data)
             size = max(size, weight)
             structs.append('btree 0x%x.%x' % (block, trunk))
-        return 'reg %s' % ', '.join(it.chain(['%d' % size], structs))
+        return '%s %s' % (
+            'scratch' if tag == TAG_SCRATCH else 'reg',
+            ', '.join(it.chain(['%d' % size], structs)))
 
     elif tag == TAG_DIR:
         # read the did
@@ -1979,6 +1983,9 @@ def main(disk, mroots=None, *,
                         # skip bookmarks
                         if tag == TAG_BOOKMARK:
                             continue
+                        # skip scratch files
+                        if tag == TAG_SCRATCH:
+                            continue
                         # skip grmed entries
                         if (max(mbid-max(mw-1, 0), 0), rid) in gstate.grm:
                             continue
@@ -2027,7 +2034,9 @@ def main(disk, mroots=None, *,
                     print('%s%12s %*s %-*s  %s%s%s' % (
                         '\x1b[31m' if color and not grmed and notes
                             else '\x1b[90m'
-                                if color and (grmed or tag == TAG_BOOKMARK)
+                                if color and (grmed
+                                    or tag == TAG_BOOKMARK
+                                    or tag == TAG_SCRATCH)
                             else '',
                         '{%s}:' % ','.join('%04x' % block
                             for block in it.chain([mdir.block],
@@ -2043,7 +2052,10 @@ def main(disk, mroots=None, *,
                         frepr(mdir, rid, tag),
                         ' (%s)' % ', '.join(notes) if notes else '',
                         '\x1b[m' if color and (
-                                notes or grmed or tag == TAG_BOOKMARK)
+                                notes
+                                    or grmed
+                                    or tag == TAG_BOOKMARK
+                                    or tag == TAG_SCRATCH)
                             else ''))
                     pmbid = mbid
 
@@ -2087,7 +2099,8 @@ def main(disk, mroots=None, *,
                                         line))
 
                     # print file contents?
-                    if tag == TAG_REG and args.get('structs'):
+                    if ((tag == TAG_REG or tag == TAG_SCRATCH)
+                            and args.get('structs')):
                         # inlined sprout?
                         done, rid_, tag_, w_, j, d, data, _ = mdir.lookup(
                             rid, TAG_DATA)

scripts/dbgmtree.py

@@ -24,6 +24,7 @@ TAG_GRMDELTA        = 0x0100
 TAG_NAME            = 0x0200
 TAG_REG             = 0x0201
 TAG_DIR             = 0x0202
+TAG_SCRATCH         = 0x0203
 TAG_BOOKMARK        = 0x0204
 TAG_STRUCT          = 0x0300
 TAG_DATA            = 0x0300
@@ -202,6 +203,7 @@ def tagrepr(tag, w, size, off=None):
             'name' if (tag & 0xfff) == TAG_NAME
                 else 'reg' if (tag & 0xfff) == TAG_REG
                 else 'dir' if (tag & 0xfff) == TAG_DIR
+                else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
                 else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
                 else 'name 0x%02x' % (tag & 0xff),
             ' w%d' % w if w else '',

scripts/dbgrbyd.py

@@ -33,6 +33,7 @@ TAG_GRMDELTA        = 0x0100
 TAG_NAME            = 0x0200
 TAG_REG             = 0x0201
 TAG_DIR             = 0x0202
+TAG_SCRATCH         = 0x0203
 TAG_BOOKMARK        = 0x0204
 TAG_STRUCT          = 0x0300
 TAG_DATA            = 0x0300
@@ -189,6 +190,7 @@ def tagrepr(tag, w, size, off=None):
             'name' if (tag & 0xfff) == TAG_NAME
                 else 'reg' if (tag & 0xfff) == TAG_REG
                 else 'dir' if (tag & 0xfff) == TAG_DIR
+                else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
                 else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
                 else 'name 0x%02x' % (tag & 0xff),
             ' w%d' % w if w else '',

tests/test_fscratch.toml

@@ -0,0 +1,341 @@
+# Test scratch files and their various use cases
+after = 'test_fwrite'
+
+
+# Some specific tests
+[cases.test_fscratch_create]
+defines.SIZE = '4*BLOCK_SIZE'
+defines.CHUNK = 64
+defines.SYNC = [false, true]
+code = '''
+    lfs_t lfs;
+    lfsr_format(&lfs, CFG) => 0;
+    lfsr_mount(&lfs, CFG) => 0;
+
+    // create a file
+    lfsr_file_t file;
+    lfsr_file_open(&lfs, &file, "gello",
+            LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0;
+
+    // check that the file doesn't _really_ exist
+    lfs_t lfs_;
+    lfsr_mount(&lfs_, CFG) => 0;
+    // via stat
+    struct lfs_info info;
+    lfsr_stat(&lfs_, "gello", &info) => LFS_ERR_NOENT;
+    // via readdir
+    lfsr_dir_t dir_;
+    lfsr_dir_open(&lfs_, &dir_, "/") => 0;
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, ".") == 0);
+    assert(info.type == LFS_TYPE_DIR);
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, "..") == 0);
+    assert(info.type == LFS_TYPE_DIR);
+    lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
+    lfsr_dir_close(&lfs_, &dir_) => 0;
+    // via open
+    lfsr_file_t file_;
+    lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => LFS_ERR_NOENT;
+    lfsr_unmount(&lfs_) => 0;
+
+    // write to the file
+    uint32_t prng = 42;
+    for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+        uint8_t wbuf[CHUNK];
+        for (lfs_size_t j = 0; j < CHUNK; j++) {
+            wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
+        }
+        lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK;
+
+        // check that the file still doesn't _really_ exist
+        lfsr_mount(&lfs_, CFG) => 0;
+        // via stat
+        lfsr_stat(&lfs_, "gello", &info) => LFS_ERR_NOENT;
+        // via readdir
+        lfsr_dir_open(&lfs_, &dir_, "/") => 0;
+        lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+        assert(strcmp(info.name, ".") == 0);
+        assert(info.type == LFS_TYPE_DIR);
+        lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+        assert(strcmp(info.name, "..") == 0);
+        assert(info.type == LFS_TYPE_DIR);
+        lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
+        lfsr_dir_close(&lfs_, &dir_) => 0;
+        // via open
+        lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => LFS_ERR_NOENT;
+        lfsr_unmount(&lfs_) => 0;
+    }
+    
+    if (SYNC) {
+        // sync the file
+        lfsr_file_sync(&lfs, &file) => 0;
+
+        // now it should show up
+        lfsr_mount(&lfs_, CFG) => 0;
+        // via stat
+        lfsr_stat(&lfs_, "gello", &info) => 0;
+        assert(strcmp(info.name, "gello") == 0);
+        assert(info.type == LFS_TYPE_REG);
+        assert(info.size == SIZE);
+        // via readdir
+        lfsr_dir_open(&lfs_, &dir_, "/") => 0;
+        lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+        assert(strcmp(info.name, ".") == 0);
+        assert(info.type == LFS_TYPE_DIR);
+        lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+        assert(strcmp(info.name, "..") == 0);
+        assert(info.type == LFS_TYPE_DIR);
+        lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+        assert(strcmp(info.name, "gello") == 0);
+        assert(info.type == LFS_TYPE_REG);
+        assert(info.size == SIZE);
+        lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
+        lfsr_dir_close(&lfs_, &dir_) => 0;
+        // via open
+        lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0;
+        uint32_t prng_ = 42;
+        for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+            uint8_t wbuf[CHUNK];
+            for (lfs_size_t j = 0; j < CHUNK; j++) {
+                wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26);
+            }
+            uint8_t rbuf[CHUNK];
+            lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK;
+            assert(memcmp(wbuf, rbuf, CHUNK) == 0);
+        }
+        lfsr_file_close(&lfs_, &file_) => 0;
+        lfsr_unmount(&lfs_) => 0;
+    }
+
+    // close the file
+    lfsr_file_close(&lfs, &file) => 0;
+
+    // now it should show up
+    lfsr_mount(&lfs_, CFG) => 0;
+    // via stat
+    lfsr_stat(&lfs_, "gello", &info) => 0;
+    assert(strcmp(info.name, "gello") == 0);
+    assert(info.type == LFS_TYPE_REG);
+    assert(info.size == SIZE);
+    // via readdir
+    lfsr_dir_open(&lfs_, &dir_, "/") => 0;
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, ".") == 0);
+    assert(info.type == LFS_TYPE_DIR);
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, "..") == 0);
+    assert(info.type == LFS_TYPE_DIR);
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, "gello") == 0);
+    assert(info.type == LFS_TYPE_REG);
+    assert(info.size == SIZE);
+    lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
+    lfsr_dir_close(&lfs_, &dir_) => 0;
+    // via open
+    lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0;
+    uint32_t prng_ = 42;
+    for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+        uint8_t wbuf[CHUNK];
+        for (lfs_size_t j = 0; j < CHUNK; j++) {
+            wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26);
+        }
+        uint8_t rbuf[CHUNK];
+        lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK;
+        assert(memcmp(wbuf, rbuf, CHUNK) == 0);
+    }
+    lfsr_file_close(&lfs_, &file_) => 0;
+    lfsr_unmount(&lfs_) => 0;
+
+    lfsr_unmount(&lfs) => 0;
+
+    // should still be there
+    lfsr_mount(&lfs_, CFG) => 0;
+    // via stat
+    lfsr_stat(&lfs_, "gello", &info) => 0;
+    assert(strcmp(info.name, "gello") == 0);
+    assert(info.type == LFS_TYPE_REG);
+    assert(info.size == SIZE);
+    // via readdir
+    lfsr_dir_open(&lfs_, &dir_, "/") => 0;
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, ".") == 0);
+    assert(info.type == LFS_TYPE_DIR);
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, "..") == 0);
+    assert(info.type == LFS_TYPE_DIR);
+    lfsr_dir_read(&lfs_, &dir_, &info) => 0;
+    assert(strcmp(info.name, "gello") == 0);
+    assert(info.type == LFS_TYPE_REG);
+    assert(info.size == SIZE);
+    lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
+    lfsr_dir_close(&lfs_, &dir_) => 0;
+    // via open
+    lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0;
+    prng_ = 42;
+    for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+        uint8_t wbuf[CHUNK];
+        for (lfs_size_t j = 0; j < CHUNK; j++) {
+            wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26);
+        }
+        uint8_t rbuf[CHUNK];
+        lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK;
+        assert(memcmp(wbuf, rbuf, CHUNK) == 0);
+    }
+    lfsr_file_close(&lfs_, &file_) => 0;
+    lfsr_unmount(&lfs_) => 0;
+'''
+
+[cases.test_fscratch_create_pl]
+defines.SIZE = 'BLOCK_SIZE'
+defines.CHUNK = 64
+reentrant = true
+code = '''
+    // format once per test
+    lfs_t lfs;
+    int err = lfsr_mount(&lfs, CFG);
+    if (err) {
+        lfsr_format(&lfs, CFG) => 0;
+        lfsr_mount(&lfs, CFG) => 0;
+    }
+
+    // create a file
+    //
+    // note the excl flag
+    lfsr_file_t file;
+    err = lfsr_file_open(&lfs, &file, "gello",
+            LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL);
+    assert(!err || err == LFS_ERR_EXIST);
+
+    if (err != LFS_ERR_EXIST) {
+        // write to the file
+        uint32_t prng = 42;
+        for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+            uint8_t wbuf[CHUNK];
+            for (lfs_size_t j = 0; j < CHUNK; j++) {
+                wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
+            }
+            lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK;
+        }
+        
+        // close the file
+        lfsr_file_close(&lfs, &file) => 0;
+    }
+
+    // we should be able to read the file now
+    lfsr_file_open(&lfs, &file, "gello", LFS_O_RDONLY) => 0;
+    uint32_t prng = 42;
+    for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+        uint8_t wbuf[CHUNK];
+        for (lfs_size_t j = 0; j < CHUNK; j++) {
+            wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
+        }
+        uint8_t rbuf[CHUNK];
+        lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
+        assert(memcmp(wbuf, rbuf, CHUNK) == 0);
+    }
+    lfsr_file_close(&lfs, &file) => 0;
+
+    // and after remounting
+    lfsr_unmount(&lfs) => 0;
+    lfsr_mount(&lfs, CFG) => 0;
+
+    lfsr_file_open(&lfs, &file, "gello", LFS_O_RDONLY) => 0;
+    prng = 42;
+    for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+        uint8_t wbuf[CHUNK];
+        for (lfs_size_t j = 0; j < CHUNK; j++) {
+            wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
+        }
+        uint8_t rbuf[CHUNK];
+        lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
+        assert(memcmp(wbuf, rbuf, CHUNK) == 0);
+    }
+    lfsr_file_close(&lfs, &file) => 0;
+
+    lfsr_unmount(&lfs) => 0;
+'''
+
+[cases.test_fscratch_create_many_pl]
+defines.SIZE = 64
+defines.CHUNK = 8
+defines.N = 128
+reentrant = true
+code = '''
+    // format once per test
+    lfs_t lfs;
+    int err = lfsr_mount(&lfs, CFG);
+    if (err) {
+        lfsr_format(&lfs, CFG) => 0;
+        lfsr_mount(&lfs, CFG) => 0;
+    }
+
+    // create N files
+    for (lfs_size_t j = 0; j < N; j++) {
+        // note the excl flag
+        char name[256];
+        sprintf(name, "gello%03x", j);
+        lfsr_file_t file;
+        err = lfsr_file_open(&lfs, &file, name,
+                LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL);
+        assert(!err || err == LFS_ERR_EXIST);
+
+        if (err != LFS_ERR_EXIST) {
+            // write to the file
+            uint32_t prng = 42 + j;
+            for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+                uint8_t wbuf[CHUNK];
+                for (lfs_size_t j = 0; j < CHUNK; j++) {
+                    wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
+                }
+                lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK;
+            }
+            
+            // close the file
+            lfsr_file_close(&lfs, &file) => 0;
+        }
+    }
+
+    // we should be able to read the files now
+    for (lfs_size_t j = 0; j < N; j++) {
+        char name[256];
+        sprintf(name, "gello%03x", j);
+        lfsr_file_t file;
+        lfsr_file_open(&lfs, &file, name, LFS_O_RDONLY) => 0;
+        uint32_t prng = 42 + j;
+        for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+            uint8_t wbuf[CHUNK];
+            for (lfs_size_t j = 0; j < CHUNK; j++) {
+                wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
+            }
+            uint8_t rbuf[CHUNK];
+            lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
+            assert(memcmp(wbuf, rbuf, CHUNK) == 0);
+        }
+        lfsr_file_close(&lfs, &file) => 0;
+    }
+
+    // and after remounting
+    lfsr_unmount(&lfs) => 0;
+    lfsr_mount(&lfs, CFG) => 0;
+
+    for (lfs_size_t j = 0; j < N; j++) {
+        char name[256];
+        sprintf(name, "gello%03x", j);
+        lfsr_file_t file;
+        lfsr_file_open(&lfs, &file, name, LFS_O_RDONLY) => 0;
+        uint32_t prng = 42 + j;
+        for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
+            uint8_t wbuf[CHUNK];
+            for (lfs_size_t j = 0; j < CHUNK; j++) {
+                wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
+            }
+            uint8_t rbuf[CHUNK];
+            lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
+            assert(memcmp(wbuf, rbuf, CHUNK) == 0);
+        }
+        lfsr_file_close(&lfs, &file) => 0;
+    }
+
+    lfsr_unmount(&lfs) => 0;
+'''