From ba505c2a376813421b2ced98fb04abd7a97c257b Mon Sep 17 00:00:00 2001 From: Christopher Haster Date: Fri, 12 Jan 2024 12:13:43 -0600 Subject: [PATCH] Implemented scratch file basics "Scratch files" are a new file type added to solve the zero-sized file problem. Though they have a few other uses that may be quite valuable. The "zero-sized file problem" is a common surprise for users, where what seems like a simple file create+write operation: lfs_file_open(&lfs, &file, "hi", LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL); lfs_file_write(&lfs, &file, "hello!", strlen("hello!")); lfs_file_close(&lfs, &file); Can end up create a zero-sized file under powerloss, breaking user assumptions and their code. The tricky thing is that this is actually correct behavior as defined by POSIX. `open` with O_CREAT creats a file entry immediately, which is initially zero-sized. And the fact that power can be lost between `open` and `close` isn't really avoidable. But this is a common enough footgun that it's probably worth deviating from POSIX here. But how to avoid zero-sized files exactly? First thought: Delay the file creation until sync/close, tracking uncreated files in-device until then. This solves the problem and avoids any intermediary state if we lose power, but came with a number of headaches: 1. Since we delay file creation, we don't immediately write the filename to disk on open. This implies we need to keep the filename allocated in RAM until the first sync/close call. The requirement to keep the filename allocated for new files until first sync/close could be added to open, and with the option to call sync immediately to save the filename (and accept the risk of zero-sized files), I don't think it would be _that_ bad of an API. But it would still be pretty bad. Extra bad because 1. there's no way to warn on misuse at compile-time, 2. use-after-free bugs have a tendency to go unnoticed annoyingly often, 3. it's a regression from the previous API, and 4. who the heck reads the more-or-less same `open` documentation for every filesystem they adopt. 2. Without an allocated mid, tracking files internally gets a lot harder. The best option I could think of was to keep the opened-file linked-list sorted by mid + (in-device) file name. This did not feel like a great solutiona and was going to add more code cost. 3. Handling mdir splits containing uncreated files adds another headache. Complicated lfsr_mdir_estimate further as it needs to decide in which mdir the uncreated files will end up, and potentially split on a filename that isn't even created yet. 4. Since the number of uncreated files can be potentially unbounded, you can't prevent an mdir from filling up with only uncreated files. On disk this ends up looking like an "empty" mdir, which need specially handling in littlefs to reclaim after powerloss. Support for empty mdirs -- the orphaned mdir scan -- was already added earlier. We already scan each mdir to build gstate, so it doesn't really add much cost. Notice that last bullet point? We already scan each mdir during mount. Why not, instead of scanning for orphaned mdirs, scan for orphaned files? So this leads to the idea of "scratch files". Instead of actually delaying file creation, fake it. Create a scratch file during open, and on the first sync/close, convert it to a regular file. If we lose power, scan for scratch files during mount, and remove them on first write. Some tradeoffs: 1. The orphan scan for scratch files is a bit more expensive than for mdirs on storage with large block sizes. We need to look at each file entry vs just each mdir, which pushed the runtime up to O(BlogB) vs O(B). Though if you also consider large mtrees, the worst case is still O(nlogn). 2. Creating intermediate scratch files adds another commit to file creation. This is probably not a big issue for flash, but may be more of a concern on devices with large prog sizes. 3. Scratch files complicate unrelated mkdir/rename/etc code a bit, since we need to consider what happens when the dest is a scratch file. But the end result is simple. And simple is good. Both for implementation headaches, and code size. Even if the on-disk state is conceptually more complicated. You may have noticed these scratch files are basically isomorphic to just setting an "uncreated" flag on the file, and that's true. There may have been a simpler route to end up with the design, but hey, as long as it works. As a plus, scratch files present a solution for a couple other things: 1. Removing an open file can become a scratch file until closed. 2. Scratch files can be used as temporary files. Open a file with O_DESYNC and never call sync and you have yourself a temporary file. Maybe in the future we should add O_TMPFILE to avoid the need for unique filenames, but that is low priority. --- lfs.c | 301 +++++++++++++++++++++------------- lfs.h | 1 + scripts/dbgbmap.py | 1 + scripts/dbgbtree.py | 2 + scripts/dbglfs.py | 23 ++- scripts/dbgmtree.py | 2 + scripts/dbgrbyd.py | 2 + tests/test_fscratch.toml | 341 +++++++++++++++++++++++++++++++++++++++ 8 files changed, 558 insertions(+), 115 deletions(-) create mode 100644 tests/test_fscratch.toml diff --git a/lfs.c b/lfs.c index bb74f5d1..eb3df0f6 100644 --- a/lfs.c +++ b/lfs.c @@ -617,6 +617,7 @@ enum lfsr_tag { LFSR_TAG_NAME = 0x0200, LFSR_TAG_REG = 0x0201, LFSR_TAG_DIR = 0x0202, + LFSR_TAG_SCRATCH = 0x0203, LFSR_TAG_BOOKMARK = 0x0204, // struct tags @@ -6939,6 +6940,11 @@ static int lfsr_mtree_pathlookup(lfs_t *lfs, const char *path, return LFS_ERR_NOENT; } + // pretend scratch files don't exist + if (tag == LFSR_TAG_SCRATCH) { + return LFS_ERR_NOENT; + } + // go on to next name name += name_size; next:; @@ -7759,9 +7765,6 @@ static int lfsr_mountinited(lfs_t *lfs) { // traverse the mtree rooted at mroot 0x{1,0} // - // note that lfsr_traversal_next will update our mroot/mtree - // based on what mroots it finds - // // we do validate btree inner nodes here, how can we trust our // mdirs are valid if we haven't checked the btree inner nodes at // least once? @@ -7796,17 +7799,6 @@ static int lfsr_mountinited(lfs_t *lfs) { *lfsr_mdir_mptr(&tinfo.u.mdir), lfsr_mweight(lfs)); } - - // found an empty non-mroot mdir? this should only happen - // if we lost power - if (tinfo.u.mdir.rbyd.weight == 0) { - LFS_DEBUG("Found orphaned mdir %"PRId32" " - "0x{%"PRIx32",%"PRIx32"}", - tinfo.u.mdir.mid >> lfs->mbits, - tinfo.u.mdir.rbyd.blocks[0], - tinfo.u.mdir.rbyd.blocks[1]); - lfs->hasorphans = true; - } } // collect any gdeltas from this mdir @@ -7815,6 +7807,27 @@ static int lfsr_mountinited(lfs_t *lfs) { return err; } + // check for any scratch files + for (lfs_size_t i = 0; + i < (lfs_size_t)tinfo.u.mdir.rbyd.weight; + i++) { + err = lfsr_mdir_lookup(lfs, &tinfo.u.mdir, + i, LFSR_TAG_SCRATCH, + NULL); + if (err && err != LFS_ERR_NOENT) { + return err; + } + + // found a scratch file? these must be orphaned + if (err != LFS_ERR_NOENT) { + LFS_DEBUG("Found orphaned scratch file " + "%"PRId32".%"PRId32, + lfsr_mid_bid(lfs, tinfo.u.mdir.mid) >> lfs->mbits, + i); + lfs->hasorphans = true; + } + } + // found an mtree inner-node? } else if (tinfo.tag == LFSR_TAG_BRANCH) { // found the root of the mtree? keep track of this @@ -8157,44 +8170,54 @@ static int lfsr_fs_fixgrm(lfs_t *lfs) { } static int lfsr_fs_fixorphans(lfs_t *lfs) { - // traverse the filesystem and drop any orphaned mdirs + // traverse the filesystem and remove any orphaned scratch files // // note this never takes longer than lfsr_mount // - lfsr_traversal_t traversal = LFSR_TRAVERSAL(0); + lfsr_mdir_t mdir; + int err = lfsr_mtree_lookup(lfs, 0, &mdir); + if (err) { + LFS_ASSERT(err != LFS_ERR_NOENT); + return err; + } + while (true) { - lfsr_tinfo_t tinfo; - int err = lfsr_traversal_read(lfs, &traversal, &tinfo); - if (err) { - if (err == LFS_ERR_NOENT) { - break; - } + // are we a scratch file? + err = lfsr_mdir_lookup(lfs, &mdir, mdir.mid, LFSR_TAG_SCRATCH, + NULL); + if (err && err != LFS_ERR_NOENT) { return err; } - // found an orphaned mdir? drop - if (tinfo.tag == LFSR_TAG_MDIR - && tinfo.u.mdir.mid != -1 - && tinfo.u.mdir.rbyd.weight == 0) { - err = lfsr_mdir_drop(lfs, &tinfo.u.mdir); + if (err != LFS_ERR_NOENT) { + // remove scratch file + err = lfsr_mdir_commit(lfs, &mdir, LFSR_ATTRS( + LFSR_ATTR(mdir.mid, RM, -1, NULL()))); if (err) { return err; } - // TODO should we have a function for this? - // TODO should traversals be "opened" and updated by - // lfsr_mdir_commit/drop? - // - // dropping an orphan changes our mtree, we need to partially - // invalidate out traversal - LFS_ASSERT(traversal.state == LFSR_TRAVERSAL_MDIR); - traversal.state = LFSR_TRAVERSAL_MDIR; - traversal.u.mtraversal.bid -= lfsr_mweight(lfs); - traversal.u.mtraversal.rid = traversal.u.mtraversal.bid; - traversal.u.mtraversal.branch = lfs->mtree.u.btree; + // seek in case our mdir was dropped + err = lfsr_mtree_seek(lfs, &mdir, 0); + if (err) { + if (err == LFS_ERR_NOENT) { + break; + } + return err; + } + } else { + // lookup next entry + err = lfsr_mtree_seek(lfs, &mdir, 1); + if (err) { + if (err == LFS_ERR_NOENT) { + break; + } + return err; + } } } + lfs->hasorphans = false; return 0; } @@ -8218,14 +8241,13 @@ static int lfsr_fs_preparemutation(lfs_t *lfs) { lfs_alloc_ckpoint(lfs); } - // fix orphaned mdirs + // fix orphaned scratch files // - // this must happen after fixgrm, since dropping mdirs risks outdating - // the grm, fixgrm can also create temporary orphans, but it should - // immediately clean them up + // this must happen after fixgrm, since removing scratch files risks + // outdating the grm // if (lfs->hasorphans) { - LFS_DEBUG("Fixing orphaned mdirs..."); + LFS_DEBUG("Fixing orphaned scratch files..."); pl = true; int err = lfsr_fs_fixorphans(lfs); @@ -8608,36 +8630,16 @@ int lfsr_rename(lfs_t *lfs, const char *old_path, const char *new_path) { return lfsr_fs_fixgrm(lfs); } -// common stat function once we have an mdir -static int lfsr_mdir_stat(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_mid_t mid, - lfsr_sdid_t did, struct lfs_info *info) { - // lookup our name tag - lfsr_tag_t tag; - lfsr_data_t data; - int err = lfsr_mdir_lookupwide(lfs, mdir, mid, LFSR_TAG_NAME, - &tag, &data); - if (err) { - return err; - } - - // get our did - lfsr_did_t did_; - err = lfsr_data_readleb128(lfs, &data, (int32_t*)&did_); - if (err) { - return err; - } - - // did mismatch? this terminates dir reads - if (did != -1 && did_ != (lfsr_did_t)did) { - return LFS_ERR_NOENT; - } - +// this just populates the info struct based on what we found +static int lfsr_stat_(lfs_t *lfs, const lfsr_mdir_t *mdir, + lfsr_tag_t tag, lfsr_data_t name, + struct lfs_info *info) { // get file type from the tag info->type = lfsr_tag_subtype(tag); - // get file name from the name entry - LFS_ASSERT(lfsr_data_size(&data) <= LFS_NAME_MAX); - lfs_ssize_t name_size = lfsr_data_read(lfs, &data, + // read the file name + LFS_ASSERT(lfsr_data_size(&name) <= LFS_NAME_MAX); + lfs_ssize_t name_size = lfsr_data_read(lfs, &name, info->name, LFS_NAME_MAX); if (name_size < 0) { return name_size; @@ -8651,7 +8653,7 @@ static int lfsr_mdir_stat(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_mid_t mid, // inlined? lfsr_tag_t tag; lfsr_data_t data; - err = lfsr_mdir_lookupnext(lfs, mdir, mid, LFSR_TAG_DATA, + int err = lfsr_mdir_lookupnext(lfs, mdir, mdir->mid, LFSR_TAG_DATA, &tag, &data); if (err && err != LFS_ERR_NOENT) { return err; @@ -8697,7 +8699,9 @@ int lfsr_stat(lfs_t *lfs, const char *path, struct lfs_info *info) { } // fill out our info struct - return lfsr_mdir_stat(lfs, &mdir, mdir.mid, -1, info); + return lfsr_stat_(lfs, &mdir, + tag, LFSR_DATA_BUF(name, name_size), + info); } int lfsr_dir_open(lfs_t *lfs, lfsr_dir_t *dir, const char *path) { @@ -8779,22 +8783,50 @@ int lfsr_dir_read(lfs_t *lfs, lfsr_dir_t *dir, struct lfs_info *info) { return err; } - // fill out our info struct - // - // this will return LFS_ERR_NOENT if our dids mismatch - err = lfsr_mdir_stat(lfs, &dir->mdir, dir->mdir.mid, dir->did, info); - if (err) { - return err; - } + while (true) { + // lookup the next name tag + lfsr_tag_t tag; + lfsr_data_t data; + err = lfsr_mdir_lookupwide(lfs, &dir->mdir, + dir->mdir.mid, LFSR_TAG_NAME, + &tag, &data); + if (err) { + return err; + } - // eagerly look up the next entry - err = lfsr_mtree_seek(lfs, &dir->mdir, 1); - if (err && err != LFS_ERR_NOENT) { - return err; - } - dir->pos += 1; + // get the did + lfsr_did_t did; + err = lfsr_data_readleb128(lfs, &data, (int32_t*)&did); + if (err) { + return err; + } - return 0; + // did mismatch? this terminates the dir read + if (did != dir->did) { + return LFS_ERR_NOENT; + } + + // skip scratch files, we pretend these don't exist + if (tag != LFSR_TAG_SCRATCH) { + // fill out our info struct + err = lfsr_stat_(lfs, &dir->mdir, tag, data, + info); + if (err) { + return err; + } + } + + // eagerly look up the next entry + err = lfsr_mtree_seek(lfs, &dir->mdir, 1); + if (err && err != LFS_ERR_NOENT) { + return err; + } + dir->pos += 1; + + if (tag != LFSR_TAG_SCRATCH) { + return 0; + } + } } int lfsr_dir_seek(lfs_t *lfs, lfsr_dir_t *dir, lfs_soff_t off) { @@ -8960,6 +8992,10 @@ static inline bool lfsr_f_isunsync(uint32_t flags) { return flags & LFS_F_UNSYNC; } +static inline bool lfsr_f_isuncreat(uint32_t flags) { + return flags & LFS_F_UNCREAT; +} + static inline lfs_off_t lfsr_file_size_(const lfsr_file_t *file) { return lfs_max32( file->buffer_pos + file->buffer_size, @@ -8977,6 +9013,15 @@ int lfsr_file_opencfg(lfs_t *lfs, lfsr_file_t *file, const struct lfs_file_config *cfg) { // don't allow the forbidden mode! LFS_ASSERT((flags & 3) != 3); + // these flags require a writable file + LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_iscreat(flags)); + LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_isexcl(flags)); + LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_istrunc(flags)); + LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_isappend(flags)); + // these flags are internal and shouldn't be provided by the user + LFS_ASSERT(!lfsr_f_isunflush(flags)); + LFS_ASSERT(!lfsr_f_isunsync(flags)); + LFS_ASSERT(!lfsr_f_isuncreat(flags)); if (!lfsr_o_isrdonly(flags)) { // prepare our filesystem for writing @@ -9018,20 +9063,17 @@ int lfsr_file_opencfg(lfs_t *lfs, lfsr_file_t *file, return LFS_ERR_NAMETOOLONG; } - // create our entry - // - // note this risks creating a zero-length file if we lose power here, - // but it's the only way for us to save the file name. - // - // TODO or is it? ;) + // create a scratch entry, this reserves the mid until first sync err = lfsr_mdir_commit(lfs, &file->mdir, LFSR_ATTRS( LFSR_ATTR(file->mdir.mid, - REG, +1, CAT( + SCRATCH, +1, CAT( LFSR_DATA_LEB128(did), LFSR_DATA_BUF(name, name_size))))); if (err) { return err; } + file->flags |= LFS_F_UNSYNC | LFS_F_UNCREAT; + } else { if (lfsr_o_isexcl(flags)) { // oh, we really wanted to create a new entry @@ -10765,23 +10807,59 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) { lfs_alloc_ckpoint(lfs); // commit our file's metadata + lfsr_attr_t attrs[2]; + lfs_size_t attr_count = 0; uint8_t buf[LFSR_BTREE_DSIZE]; - err = lfsr_mdir_commit(lfs, &file->mdir, LFSR_ATTRS( - (lfsr_f_isunflush(file->flags) && file->buffer_size == 0) - ? LFSR_ATTR(file->mdir.mid, - WIDE(RM(STRUCT)), 0, - NULL()) - : (lfsr_f_isunflush(file->flags)) - ? LFSR_ATTR(file->mdir.mid, - WIDE(DATA), 0, - BUF(file->buffer, file->buffer_size)) - : (lfsr_ftree_isbshrub(&file->mdir, &file->ftree)) - ? LFSR_ATTR(file->mdir.mid, - WIDE(SHRUBTRUNK), 0, - SHRUBTRUNK(&file->ftree_.u.bshrub)) - : LFSR_ATTR(file->mdir.mid, - WIDE(BTREE), 0, - FROMBTREE(&file->ftree.u.btree, buf)))); + lfs_size_t buf_size = 0; + + // not created yet? need to convert scratch file to normal file + if (lfsr_f_isuncreat(file->flags)) { + lfsr_data_t data; + err = lfsr_mdir_lookup(lfs, &file->mdir, + file->mdir.mid, LFSR_TAG_SCRATCH, + &data); + if (err) { + // we must have a scratch file at this point + LFS_ASSERT(err != LFS_ERR_NOENT); + goto failed; + } + + attrs[attr_count++] = LFSR_ATTR(file->mdir.mid, + WIDE(REG), 0, DATA(data)); + } + + // commit the file state + + // null? no attr? + if (lfsr_f_isunflush(file->flags) && file->buffer_size == 0) { + attrs[attr_count++] = LFSR_ATTR(file->mdir.mid, + WIDE(RM(STRUCT)), 0, + NULL()); + // small file inlined in mdir? + } else if (lfsr_f_isunflush(file->flags)) { + attrs[attr_count++] = LFSR_ATTR(file->mdir.mid, + WIDE(DATA), 0, + BUF(file->buffer, file->buffer_size)); + // bshrub? + } else if (lfsr_ftree_isbshrub(&file->mdir, &file->ftree)) { + attrs[attr_count++] = LFSR_ATTR(file->mdir.mid, + WIDE(SHRUBTRUNK), 0, + SHRUBTRUNK(&file->ftree_.u.bshrub)); + // btree? + } else if (lfsr_ftree_isbtree(&file->mdir, &file->ftree)) { + attrs[attr_count++] = LFSR_ATTR(file->mdir.mid, + WIDE(BTREE), 0, + FROMBTREE(&file->ftree.u.btree, &buf[buf_size])); + buf_size += LFSR_BTREE_DSIZE; + } else { + LFS_UNREACHABLE(); + } + + LFS_ASSERT(attr_count <= sizeof(attrs)/sizeof(lfsr_attr_t)); + LFS_ASSERT(buf_size <= sizeof(buf)); + + err = lfsr_mdir_commit(lfs, &file->mdir, + attrs, attr_count); if (err) { goto failed; } @@ -10796,6 +10874,9 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) { && file_->mdir.mid == file->mdir.mid // don't double update && file_ != file) { + // notify all files of creation + file_->flags &= ~LFS_F_UNCREAT; + // mark desynced files an unsynced if (lfsr_o_isdesync(file_->flags)) { file_->flags |= LFS_F_UNSYNC; @@ -10818,7 +10899,7 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) { } // mark as synced - file->flags &= ~LFS_F_UNSYNC & ~LFS_O_DESYNC; + file->flags &= ~LFS_F_UNSYNC & ~LFS_F_UNCREAT & ~LFS_O_DESYNC; return 0; failed:; diff --git a/lfs.h b/lfs.h index 2e276b8c..d3ff6c64 100644 --- a/lfs.h +++ b/lfs.h @@ -139,6 +139,7 @@ enum lfs_open_flags { // internally used flags LFS_F_UNFLUSH = 0x1000, // File's data does not match storage LFS_F_UNSYNC = 0x2000, // File's metadata does not match storage + LFS_F_UNCREAT = 0x4000, // File does not exist yet }; // File seek flags diff --git a/scripts/dbgbmap.py b/scripts/dbgbmap.py index 848b7553..53f8bdbb 100755 --- a/scripts/dbgbmap.py +++ b/scripts/dbgbmap.py @@ -26,6 +26,7 @@ TAG_GRMDELTA = 0x0100 TAG_NAME = 0x0200 TAG_REG = 0x0201 TAG_DIR = 0x0202 +TAG_SCRATCH = 0x0203 TAG_BOOKMARK = 0x0204 TAG_STRUCT = 0x0300 TAG_DATA = 0x0300 diff --git a/scripts/dbgbtree.py b/scripts/dbgbtree.py index df4253d1..8c0c725f 100755 --- a/scripts/dbgbtree.py +++ b/scripts/dbgbtree.py @@ -24,6 +24,7 @@ TAG_GRMDELTA = 0x0100 TAG_NAME = 0x0200 TAG_REG = 0x0201 TAG_DIR = 0x0202 +TAG_SCRATCH = 0x0203 TAG_BOOKMARK = 0x0204 TAG_STRUCT = 0x0300 TAG_DATA = 0x0300 @@ -187,6 +188,7 @@ def tagrepr(tag, w, size, off=None): 'name' if (tag & 0xfff) == TAG_NAME else 'reg' if (tag & 0xfff) == TAG_REG else 'dir' if (tag & 0xfff) == TAG_DIR + else 'scratch' if (tag & 0xfff) == TAG_SCRATCH else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK else 'name 0x%02x' % (tag & 0xff), ' w%d' % w if w else '', diff --git a/scripts/dbglfs.py b/scripts/dbglfs.py index eef7a3a7..d258df57 100755 --- a/scripts/dbglfs.py +++ b/scripts/dbglfs.py @@ -25,6 +25,7 @@ TAG_GRMDELTA = 0x0100 TAG_NAME = 0x0200 TAG_REG = 0x0201 TAG_DIR = 0x0202 +TAG_SCRATCH = 0x0203 TAG_BOOKMARK = 0x0204 TAG_STRUCT = 0x0300 TAG_DATA = 0x0300 @@ -218,6 +219,7 @@ def tagrepr(tag, w, size, off=None): 'name' if (tag & 0xfff) == TAG_NAME else 'reg' if (tag & 0xfff) == TAG_REG else 'dir' if (tag & 0xfff) == TAG_DIR + else 'scratch' if (tag & 0xfff) == TAG_SCRATCH else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK else 'name 0x%02x' % (tag & 0xff), ' w%d' % w if w else '', @@ -1179,7 +1181,7 @@ class GState: yield grepr(tag, data), tag, data def frepr(mdir, rid, tag): - if tag == TAG_REG: + if tag == TAG_REG or tag == TAG_SCRATCH: size = 0 structs = [] # inlined data? @@ -1205,7 +1207,9 @@ def frepr(mdir, rid, tag): weight, block, trunk, cksum = frombtree(data) size = max(size, weight) structs.append('btree 0x%x.%x' % (block, trunk)) - return 'reg %s' % ', '.join(it.chain(['%d' % size], structs)) + return '%s %s' % ( + 'scratch' if tag == TAG_SCRATCH else 'reg', + ', '.join(it.chain(['%d' % size], structs))) elif tag == TAG_DIR: # read the did @@ -1979,6 +1983,9 @@ def main(disk, mroots=None, *, # skip bookmarks if tag == TAG_BOOKMARK: continue + # skip scratch files + if tag == TAG_SCRATCH: + continue # skip grmed entries if (max(mbid-max(mw-1, 0), 0), rid) in gstate.grm: continue @@ -2027,7 +2034,9 @@ def main(disk, mroots=None, *, print('%s%12s %*s %-*s %s%s%s' % ( '\x1b[31m' if color and not grmed and notes else '\x1b[90m' - if color and (grmed or tag == TAG_BOOKMARK) + if color and (grmed + or tag == TAG_BOOKMARK + or tag == TAG_SCRATCH) else '', '{%s}:' % ','.join('%04x' % block for block in it.chain([mdir.block], @@ -2043,7 +2052,10 @@ def main(disk, mroots=None, *, frepr(mdir, rid, tag), ' (%s)' % ', '.join(notes) if notes else '', '\x1b[m' if color and ( - notes or grmed or tag == TAG_BOOKMARK) + notes + or grmed + or tag == TAG_BOOKMARK + or tag == TAG_SCRATCH) else '')) pmbid = mbid @@ -2087,7 +2099,8 @@ def main(disk, mroots=None, *, line)) # print file contents? - if tag == TAG_REG and args.get('structs'): + if ((tag == TAG_REG or tag == TAG_SCRATCH) + and args.get('structs')): # inlined sprout? done, rid_, tag_, w_, j, d, data, _ = mdir.lookup( rid, TAG_DATA) diff --git a/scripts/dbgmtree.py b/scripts/dbgmtree.py index c9c865a2..a29b5148 100755 --- a/scripts/dbgmtree.py +++ b/scripts/dbgmtree.py @@ -24,6 +24,7 @@ TAG_GRMDELTA = 0x0100 TAG_NAME = 0x0200 TAG_REG = 0x0201 TAG_DIR = 0x0202 +TAG_SCRATCH = 0x0203 TAG_BOOKMARK = 0x0204 TAG_STRUCT = 0x0300 TAG_DATA = 0x0300 @@ -202,6 +203,7 @@ def tagrepr(tag, w, size, off=None): 'name' if (tag & 0xfff) == TAG_NAME else 'reg' if (tag & 0xfff) == TAG_REG else 'dir' if (tag & 0xfff) == TAG_DIR + else 'scratch' if (tag & 0xfff) == TAG_SCRATCH else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK else 'name 0x%02x' % (tag & 0xff), ' w%d' % w if w else '', diff --git a/scripts/dbgrbyd.py b/scripts/dbgrbyd.py index 0438d6b8..91524d28 100755 --- a/scripts/dbgrbyd.py +++ b/scripts/dbgrbyd.py @@ -33,6 +33,7 @@ TAG_GRMDELTA = 0x0100 TAG_NAME = 0x0200 TAG_REG = 0x0201 TAG_DIR = 0x0202 +TAG_SCRATCH = 0x0203 TAG_BOOKMARK = 0x0204 TAG_STRUCT = 0x0300 TAG_DATA = 0x0300 @@ -189,6 +190,7 @@ def tagrepr(tag, w, size, off=None): 'name' if (tag & 0xfff) == TAG_NAME else 'reg' if (tag & 0xfff) == TAG_REG else 'dir' if (tag & 0xfff) == TAG_DIR + else 'scratch' if (tag & 0xfff) == TAG_SCRATCH else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK else 'name 0x%02x' % (tag & 0xff), ' w%d' % w if w else '', diff --git a/tests/test_fscratch.toml b/tests/test_fscratch.toml new file mode 100644 index 00000000..918684dc --- /dev/null +++ b/tests/test_fscratch.toml @@ -0,0 +1,341 @@ +# Test scratch files and their various use cases +after = 'test_fwrite' + + +# Some specific tests +[cases.test_fscratch_create] +defines.SIZE = '4*BLOCK_SIZE' +defines.CHUNK = 64 +defines.SYNC = [false, true] +code = ''' + lfs_t lfs; + lfsr_format(&lfs, CFG) => 0; + lfsr_mount(&lfs, CFG) => 0; + + // create a file + lfsr_file_t file; + lfsr_file_open(&lfs, &file, "gello", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0; + + // check that the file doesn't _really_ exist + lfs_t lfs_; + lfsr_mount(&lfs_, CFG) => 0; + // via stat + struct lfs_info info; + lfsr_stat(&lfs_, "gello", &info) => LFS_ERR_NOENT; + // via readdir + lfsr_dir_t dir_; + lfsr_dir_open(&lfs_, &dir_, "/") => 0; + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT; + lfsr_dir_close(&lfs_, &dir_) => 0; + // via open + lfsr_file_t file_; + lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfsr_unmount(&lfs_) => 0; + + // write to the file + uint32_t prng = 42; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26); + } + lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK; + + // check that the file still doesn't _really_ exist + lfsr_mount(&lfs_, CFG) => 0; + // via stat + lfsr_stat(&lfs_, "gello", &info) => LFS_ERR_NOENT; + // via readdir + lfsr_dir_open(&lfs_, &dir_, "/") => 0; + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT; + lfsr_dir_close(&lfs_, &dir_) => 0; + // via open + lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => LFS_ERR_NOENT; + lfsr_unmount(&lfs_) => 0; + } + + if (SYNC) { + // sync the file + lfsr_file_sync(&lfs, &file) => 0; + + // now it should show up + lfsr_mount(&lfs_, CFG) => 0; + // via stat + lfsr_stat(&lfs_, "gello", &info) => 0; + assert(strcmp(info.name, "gello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + // via readdir + lfsr_dir_open(&lfs_, &dir_, "/") => 0; + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "gello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT; + lfsr_dir_close(&lfs_, &dir_) => 0; + // via open + lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0; + uint32_t prng_ = 42; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26); + } + uint8_t rbuf[CHUNK]; + lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK; + assert(memcmp(wbuf, rbuf, CHUNK) == 0); + } + lfsr_file_close(&lfs_, &file_) => 0; + lfsr_unmount(&lfs_) => 0; + } + + // close the file + lfsr_file_close(&lfs, &file) => 0; + + // now it should show up + lfsr_mount(&lfs_, CFG) => 0; + // via stat + lfsr_stat(&lfs_, "gello", &info) => 0; + assert(strcmp(info.name, "gello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + // via readdir + lfsr_dir_open(&lfs_, &dir_, "/") => 0; + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "gello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT; + lfsr_dir_close(&lfs_, &dir_) => 0; + // via open + lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0; + uint32_t prng_ = 42; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26); + } + uint8_t rbuf[CHUNK]; + lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK; + assert(memcmp(wbuf, rbuf, CHUNK) == 0); + } + lfsr_file_close(&lfs_, &file_) => 0; + lfsr_unmount(&lfs_) => 0; + + lfsr_unmount(&lfs) => 0; + + // should still be there + lfsr_mount(&lfs_, CFG) => 0; + // via stat + lfsr_stat(&lfs_, "gello", &info) => 0; + assert(strcmp(info.name, "gello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + // via readdir + lfsr_dir_open(&lfs_, &dir_, "/") => 0; + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, ".") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "..") == 0); + assert(info.type == LFS_TYPE_DIR); + lfsr_dir_read(&lfs_, &dir_, &info) => 0; + assert(strcmp(info.name, "gello") == 0); + assert(info.type == LFS_TYPE_REG); + assert(info.size == SIZE); + lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT; + lfsr_dir_close(&lfs_, &dir_) => 0; + // via open + lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0; + prng_ = 42; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26); + } + uint8_t rbuf[CHUNK]; + lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK; + assert(memcmp(wbuf, rbuf, CHUNK) == 0); + } + lfsr_file_close(&lfs_, &file_) => 0; + lfsr_unmount(&lfs_) => 0; +''' + +[cases.test_fscratch_create_pl] +defines.SIZE = 'BLOCK_SIZE' +defines.CHUNK = 64 +reentrant = true +code = ''' + // format once per test + lfs_t lfs; + int err = lfsr_mount(&lfs, CFG); + if (err) { + lfsr_format(&lfs, CFG) => 0; + lfsr_mount(&lfs, CFG) => 0; + } + + // create a file + // + // note the excl flag + lfsr_file_t file; + err = lfsr_file_open(&lfs, &file, "gello", + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL); + assert(!err || err == LFS_ERR_EXIST); + + if (err != LFS_ERR_EXIST) { + // write to the file + uint32_t prng = 42; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26); + } + lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK; + } + + // close the file + lfsr_file_close(&lfs, &file) => 0; + } + + // we should be able to read the file now + lfsr_file_open(&lfs, &file, "gello", LFS_O_RDONLY) => 0; + uint32_t prng = 42; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26); + } + uint8_t rbuf[CHUNK]; + lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK; + assert(memcmp(wbuf, rbuf, CHUNK) == 0); + } + lfsr_file_close(&lfs, &file) => 0; + + // and after remounting + lfsr_unmount(&lfs) => 0; + lfsr_mount(&lfs, CFG) => 0; + + lfsr_file_open(&lfs, &file, "gello", LFS_O_RDONLY) => 0; + prng = 42; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26); + } + uint8_t rbuf[CHUNK]; + lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK; + assert(memcmp(wbuf, rbuf, CHUNK) == 0); + } + lfsr_file_close(&lfs, &file) => 0; + + lfsr_unmount(&lfs) => 0; +''' + +[cases.test_fscratch_create_many_pl] +defines.SIZE = 64 +defines.CHUNK = 8 +defines.N = 128 +reentrant = true +code = ''' + // format once per test + lfs_t lfs; + int err = lfsr_mount(&lfs, CFG); + if (err) { + lfsr_format(&lfs, CFG) => 0; + lfsr_mount(&lfs, CFG) => 0; + } + + // create N files + for (lfs_size_t j = 0; j < N; j++) { + // note the excl flag + char name[256]; + sprintf(name, "gello%03x", j); + lfsr_file_t file; + err = lfsr_file_open(&lfs, &file, name, + LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL); + assert(!err || err == LFS_ERR_EXIST); + + if (err != LFS_ERR_EXIST) { + // write to the file + uint32_t prng = 42 + j; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26); + } + lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK; + } + + // close the file + lfsr_file_close(&lfs, &file) => 0; + } + } + + // we should be able to read the files now + for (lfs_size_t j = 0; j < N; j++) { + char name[256]; + sprintf(name, "gello%03x", j); + lfsr_file_t file; + lfsr_file_open(&lfs, &file, name, LFS_O_RDONLY) => 0; + uint32_t prng = 42 + j; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26); + } + uint8_t rbuf[CHUNK]; + lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK; + assert(memcmp(wbuf, rbuf, CHUNK) == 0); + } + lfsr_file_close(&lfs, &file) => 0; + } + + // and after remounting + lfsr_unmount(&lfs) => 0; + lfsr_mount(&lfs, CFG) => 0; + + for (lfs_size_t j = 0; j < N; j++) { + char name[256]; + sprintf(name, "gello%03x", j); + lfsr_file_t file; + lfsr_file_open(&lfs, &file, name, LFS_O_RDONLY) => 0; + uint32_t prng = 42 + j; + for (lfs_off_t i = 0; i < SIZE; i += CHUNK) { + uint8_t wbuf[CHUNK]; + for (lfs_size_t j = 0; j < CHUNK; j++) { + wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26); + } + uint8_t rbuf[CHUNK]; + lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK; + assert(memcmp(wbuf, rbuf, CHUNK) == 0); + } + lfsr_file_close(&lfs, &file) => 0; + } + + lfsr_unmount(&lfs) => 0; +'''