Github/littlefs
1
Fork 0
Code Actions Activity

Implemented scratch file basics

Browse Source 
"Scratch files" are a new file type added to solve the zero-sized
file problem. Though they have a few other uses that may be quite
valuable.

The "zero-sized file problem" is a common surprise for users, where what
seems like a simple file create+write operation:

  lfs_file_open(&lfs, &file, "hi",
          LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL);
  lfs_file_write(&lfs, &file, "hello!", strlen("hello!"));
  lfs_file_close(&lfs, &file);

Can end up create a zero-sized file under powerloss, breaking user
assumptions and their code.

The tricky thing is that this is actually correct behavior as defined by
POSIX. `open` with O_CREAT creats a file entry immediately, which is
initially zero-sized. And the fact that power can be lost between `open`
and `close` isn't really avoidable.

But this is a common enough footgun that it's probably worth deviating
from POSIX here.

But how to avoid zero-sized files exactly? First thought: Delay the file
creation until sync/close, tracking uncreated files in-device until
then. This solves the problem and avoids any intermediary state if we
lose power, but came with a number of headaches:

1. Since we delay file creation, we don't immediately write the filename
   to disk on open. This implies we need to keep the filename allocated
   in RAM until the first sync/close call.

   The requirement to keep the filename allocated for new files until
   first sync/close could be added to open, and with the option to call
   sync immediately to save the filename (and accept the risk of
   zero-sized files), I don't think it would be _that_ bad of an API.

   But it would still be pretty bad. Extra bad because 1. there's no
   way to warn on misuse at compile-time, 2. use-after-free bugs have a
   tendency to go unnoticed annoyingly often, 3. it's a regression from
   the previous API, and 4. who the heck reads the more-or-less same
   `open` documentation for every filesystem they adopt.

2. Without an allocated mid, tracking files internally gets a lot
   harder. The best option I could think of was to keep the opened-file
   linked-list sorted by mid + (in-device) file name.

   This did not feel like a great solutiona and was going to add more
   code cost.

3. Handling mdir splits containing uncreated files adds another
   headache. Complicated lfsr_mdir_estimate further as it needs to
   decide in which mdir the uncreated files will end up, and potentially
   split on a filename that isn't even created yet.

4. Since the number of uncreated files can be potentially unbounded, you
   can't prevent an mdir from filling up with only uncreated files. On
   disk this ends up looking like an "empty" mdir, which need specially
   handling in littlefs to reclaim after powerloss.

   Support for empty mdirs -- the orphaned mdir scan -- was already
   added earlier. We already scan each mdir to build gstate, so it
   doesn't really add much cost.

Notice that last bullet point? We already scan each mdir during mount.
Why not, instead of scanning for orphaned mdirs, scan for orphaned
files?

So this leads to the idea of "scratch files". Instead of actually
delaying file creation, fake it. Create a scratch file during open, and
on the first sync/close, convert it to a regular file. If we lose power,
scan for scratch files during mount, and remove them on first write.

Some tradeoffs:

1. The orphan scan for scratch files is a bit more expensive than for
   mdirs on storage with large block sizes. We need to look at each file
   entry vs just each mdir, which pushed the runtime up to O(BlogB) vs
   O(B).

   Though if you also consider large mtrees, the worst case is still
   O(nlogn).

2. Creating intermediate scratch files adds another commit to file
   creation.

   This is probably not a big issue for flash, but may be more of a
   concern on devices with large prog sizes.

3. Scratch files complicate unrelated mkdir/rename/etc code a bit, since
   we need to consider what happens when the dest is a scratch file.

But the end result is simple. And simple is good. Both for
implementation headaches, and code size. Even if the on-disk state is
conceptually more complicated.

You may have noticed these scratch files are basically isomorphic to
just setting an "uncreated" flag on the file, and that's true. There may
have been a simpler route to end up with the design, but hey, as long as
it works.

As a plus, scratch files present a solution for a couple other things:

1. Removing an open file can become a scratch file until closed.

2. Scratch files can be used as temporary files. Open a file with
   O_DESYNC and never call sync and you have yourself a temporary file.

   Maybe in the future we should add O_TMPFILE to avoid the need for
   unique filenames, but that is low priority.
This commit is contained in:
Christopher Haster
2024-01-12 12:13:43 -06:00
parent f1697261a9
commit ba505c2a37
8 changed files with 558 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lfs.c
+191 -110
View File
@@ -617,6 +617,7 @@ enum lfsr_tag {
LFSR_TAG_NAME = 0x0200,
LFSR_TAG_REG = 0x0201,
LFSR_TAG_DIR = 0x0202,
LFSR_TAG_SCRATCH = 0x0203,
LFSR_TAG_BOOKMARK = 0x0204,
// struct tags
@@ -6939,6 +6940,11 @@ static int lfsr_mtree_pathlookup(lfs_t *lfs, const char *path,
return LFS_ERR_NOENT;
}
// pretend scratch files don't exist
if (tag == LFSR_TAG_SCRATCH) {
return LFS_ERR_NOENT;
}
// go on to next name
name += name_size;
next:;
@@ -7759,9 +7765,6 @@ static int lfsr_mountinited(lfs_t *lfs) {
// traverse the mtree rooted at mroot 0x{1,0}
//
// note that lfsr_traversal_next will update our mroot/mtree
// based on what mroots it finds
//
// we do validate btree inner nodes here, how can we trust our
// mdirs are valid if we haven't checked the btree inner nodes at
// least once?
@@ -7796,17 +7799,6 @@ static int lfsr_mountinited(lfs_t *lfs) {
*lfsr_mdir_mptr(&tinfo.u.mdir),
lfsr_mweight(lfs));
}
// found an empty non-mroot mdir? this should only happen
// if we lost power
if (tinfo.u.mdir.rbyd.weight == 0) {
LFS_DEBUG("Found orphaned mdir %"PRId32" "
"0x{%"PRIx32",%"PRIx32"}",
tinfo.u.mdir.mid >> lfs->mbits,
tinfo.u.mdir.rbyd.blocks[0],
tinfo.u.mdir.rbyd.blocks[1]);
lfs->hasorphans = true;
}
}
// collect any gdeltas from this mdir
@@ -7815,6 +7807,27 @@ static int lfsr_mountinited(lfs_t *lfs) {
return err;
}
// check for any scratch files
for (lfs_size_t i = 0;
i < (lfs_size_t)tinfo.u.mdir.rbyd.weight;
i++) {
err = lfsr_mdir_lookup(lfs, &tinfo.u.mdir,
i, LFSR_TAG_SCRATCH,
NULL);
if (err && err != LFS_ERR_NOENT) {
return err;
}
// found a scratch file? these must be orphaned
if (err != LFS_ERR_NOENT) {
LFS_DEBUG("Found orphaned scratch file "
"%"PRId32".%"PRId32,
lfsr_mid_bid(lfs, tinfo.u.mdir.mid) >> lfs->mbits,
i);
lfs->hasorphans = true;
}
}
// found an mtree inner-node?
} else if (tinfo.tag == LFSR_TAG_BRANCH) {
// found the root of the mtree? keep track of this
@@ -8157,44 +8170,54 @@ static int lfsr_fs_fixgrm(lfs_t *lfs) {
}
static int lfsr_fs_fixorphans(lfs_t *lfs) {
// traverse the filesystem and drop any orphaned mdirs
// traverse the filesystem and remove any orphaned scratch files
//
// note this never takes longer than lfsr_mount
//
lfsr_traversal_t traversal = LFSR_TRAVERSAL(0);
lfsr_mdir_t mdir;
int err = lfsr_mtree_lookup(lfs, 0, &mdir);
if (err) {
LFS_ASSERT(err != LFS_ERR_NOENT);
return err;
}
while (true) {
lfsr_tinfo_t tinfo;
int err = lfsr_traversal_read(lfs, &traversal, &tinfo);
if (err) {
if (err == LFS_ERR_NOENT) {
break;
}
// are we a scratch file?
err = lfsr_mdir_lookup(lfs, &mdir, mdir.mid, LFSR_TAG_SCRATCH,
NULL);
if (err && err != LFS_ERR_NOENT) {
return err;
}
// found an orphaned mdir? drop
if (tinfo.tag == LFSR_TAG_MDIR
&& tinfo.u.mdir.mid != -1
&& tinfo.u.mdir.rbyd.weight == 0) {
err = lfsr_mdir_drop(lfs, &tinfo.u.mdir);
if (err != LFS_ERR_NOENT) {
// remove scratch file
err = lfsr_mdir_commit(lfs, &mdir, LFSR_ATTRS(
LFSR_ATTR(mdir.mid, RM, -1, NULL())));
if (err) {
return err;
}
// TODO should we have a function for this?
// TODO should traversals be "opened" and updated by
// lfsr_mdir_commit/drop?
//
// dropping an orphan changes our mtree, we need to partially
// invalidate out traversal
LFS_ASSERT(traversal.state == LFSR_TRAVERSAL_MDIR);
traversal.state = LFSR_TRAVERSAL_MDIR;
traversal.u.mtraversal.bid -= lfsr_mweight(lfs);
traversal.u.mtraversal.rid = traversal.u.mtraversal.bid;
traversal.u.mtraversal.branch = lfs->mtree.u.btree;
// seek in case our mdir was dropped
err = lfsr_mtree_seek(lfs, &mdir, 0);
if (err) {
if (err == LFS_ERR_NOENT) {
break;
}
return err;
}
} else {
// lookup next entry
err = lfsr_mtree_seek(lfs, &mdir, 1);
if (err) {
if (err == LFS_ERR_NOENT) {
break;
}
return err;
}
}
}
lfs->hasorphans = false;
return 0;
}
@@ -8218,14 +8241,13 @@ static int lfsr_fs_preparemutation(lfs_t *lfs) {
lfs_alloc_ckpoint(lfs);
}
// fix orphaned mdirs
// fix orphaned scratch files
//
// this must happen after fixgrm, since dropping mdirs risks outdating
// the grm, fixgrm can also create temporary orphans, but it should
// immediately clean them up
// this must happen after fixgrm, since removing scratch files risks
// outdating the grm
//
if (lfs->hasorphans) {
LFS_DEBUG("Fixing orphaned mdirs...");
LFS_DEBUG("Fixing orphaned scratch files...");
pl = true;
int err = lfsr_fs_fixorphans(lfs);
@@ -8608,36 +8630,16 @@ int lfsr_rename(lfs_t *lfs, const char *old_path, const char *new_path) {
return lfsr_fs_fixgrm(lfs);
}
// common stat function once we have an mdir
static int lfsr_mdir_stat(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_mid_t mid,
lfsr_sdid_t did, struct lfs_info *info) {
// lookup our name tag
lfsr_tag_t tag;
lfsr_data_t data;
int err = lfsr_mdir_lookupwide(lfs, mdir, mid, LFSR_TAG_NAME,
&tag, &data);
if (err) {
return err;
}
// get our did
lfsr_did_t did_;
err = lfsr_data_readleb128(lfs, &data, (int32_t*)&did_);
if (err) {
return err;
}
// did mismatch? this terminates dir reads
if (did != -1 && did_ != (lfsr_did_t)did) {
return LFS_ERR_NOENT;
}
// this just populates the info struct based on what we found
static int lfsr_stat_(lfs_t *lfs, const lfsr_mdir_t *mdir,
lfsr_tag_t tag, lfsr_data_t name,
struct lfs_info *info) {
// get file type from the tag
info->type = lfsr_tag_subtype(tag);
// get file name from the name entry
LFS_ASSERT(lfsr_data_size(&data) <= LFS_NAME_MAX);
lfs_ssize_t name_size = lfsr_data_read(lfs, &data,
// read the file name
LFS_ASSERT(lfsr_data_size(&name) <= LFS_NAME_MAX);
lfs_ssize_t name_size = lfsr_data_read(lfs, &name,
info->name, LFS_NAME_MAX);
if (name_size < 0) {
return name_size;
@@ -8651,7 +8653,7 @@ static int lfsr_mdir_stat(lfs_t *lfs, lfsr_mdir_t *mdir, lfsr_mid_t mid,
// inlined?
lfsr_tag_t tag;
lfsr_data_t data;
err = lfsr_mdir_lookupnext(lfs, mdir, mid, LFSR_TAG_DATA,
int err = lfsr_mdir_lookupnext(lfs, mdir, mdir->mid, LFSR_TAG_DATA,
&tag, &data);
if (err && err != LFS_ERR_NOENT) {
return err;
@@ -8697,7 +8699,9 @@ int lfsr_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
}
// fill out our info struct
return lfsr_mdir_stat(lfs, &mdir, mdir.mid, -1, info);
return lfsr_stat_(lfs, &mdir,
tag, LFSR_DATA_BUF(name, name_size),
info);
}
int lfsr_dir_open(lfs_t *lfs, lfsr_dir_t *dir, const char *path) {
@@ -8779,22 +8783,50 @@ int lfsr_dir_read(lfs_t *lfs, lfsr_dir_t *dir, struct lfs_info *info) {
return err;
}
// fill out our info struct
//
// this will return LFS_ERR_NOENT if our dids mismatch
err = lfsr_mdir_stat(lfs, &dir->mdir, dir->mdir.mid, dir->did, info);
if (err) {
return err;
}
while (true) {
// lookup the next name tag
lfsr_tag_t tag;
lfsr_data_t data;
err = lfsr_mdir_lookupwide(lfs, &dir->mdir,
dir->mdir.mid, LFSR_TAG_NAME,
&tag, &data);
if (err) {
return err;
}
// eagerly look up the next entry
err = lfsr_mtree_seek(lfs, &dir->mdir, 1);
if (err && err != LFS_ERR_NOENT) {
return err;
}
dir->pos += 1;
// get the did
lfsr_did_t did;
err = lfsr_data_readleb128(lfs, &data, (int32_t*)&did);
if (err) {
return err;
}
return 0;
// did mismatch? this terminates the dir read
if (did != dir->did) {
return LFS_ERR_NOENT;
}
// skip scratch files, we pretend these don't exist
if (tag != LFSR_TAG_SCRATCH) {
// fill out our info struct
err = lfsr_stat_(lfs, &dir->mdir, tag, data,
info);
if (err) {
return err;
}
}
// eagerly look up the next entry
err = lfsr_mtree_seek(lfs, &dir->mdir, 1);
if (err && err != LFS_ERR_NOENT) {
return err;
}
dir->pos += 1;
if (tag != LFSR_TAG_SCRATCH) {
return 0;
}
}
}
int lfsr_dir_seek(lfs_t *lfs, lfsr_dir_t *dir, lfs_soff_t off) {
@@ -8960,6 +8992,10 @@ static inline bool lfsr_f_isunsync(uint32_t flags) {
return flags & LFS_F_UNSYNC;
}
static inline bool lfsr_f_isuncreat(uint32_t flags) {
return flags & LFS_F_UNCREAT;
}
static inline lfs_off_t lfsr_file_size_(const lfsr_file_t *file) {
return lfs_max32(
file->buffer_pos + file->buffer_size,
@@ -8977,6 +9013,15 @@ int lfsr_file_opencfg(lfs_t *lfs, lfsr_file_t *file,
const struct lfs_file_config *cfg) {
// don't allow the forbidden mode!
LFS_ASSERT((flags & 3) != 3);
// these flags require a writable file
LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_iscreat(flags));
LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_isexcl(flags));
LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_istrunc(flags));
LFS_ASSERT(!lfsr_o_isrdonly(flags) || !lfsr_o_isappend(flags));
// these flags are internal and shouldn't be provided by the user
LFS_ASSERT(!lfsr_f_isunflush(flags));
LFS_ASSERT(!lfsr_f_isunsync(flags));
LFS_ASSERT(!lfsr_f_isuncreat(flags));
if (!lfsr_o_isrdonly(flags)) {
// prepare our filesystem for writing
@@ -9018,20 +9063,17 @@ int lfsr_file_opencfg(lfs_t *lfs, lfsr_file_t *file,
return LFS_ERR_NAMETOOLONG;
}
// create our entry
//
// note this risks creating a zero-length file if we lose power here,
// but it's the only way for us to save the file name.
//
// TODO or is it? ;)
// create a scratch entry, this reserves the mid until first sync
err = lfsr_mdir_commit(lfs, &file->mdir, LFSR_ATTRS(
LFSR_ATTR(file->mdir.mid,
REG, +1, CAT(
SCRATCH, +1, CAT(
LFSR_DATA_LEB128(did),
LFSR_DATA_BUF(name, name_size)))));
if (err) {
return err;
}
file->flags |= LFS_F_UNSYNC | LFS_F_UNCREAT;
} else {
if (lfsr_o_isexcl(flags)) {
// oh, we really wanted to create a new entry
@@ -10765,23 +10807,59 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) {
lfs_alloc_ckpoint(lfs);
// commit our file's metadata
lfsr_attr_t attrs[2];
lfs_size_t attr_count = 0;
uint8_t buf[LFSR_BTREE_DSIZE];
err = lfsr_mdir_commit(lfs, &file->mdir, LFSR_ATTRS(
(lfsr_f_isunflush(file->flags) && file->buffer_size == 0)
? LFSR_ATTR(file->mdir.mid,
WIDE(RM(STRUCT)), 0,
NULL())
: (lfsr_f_isunflush(file->flags))
? LFSR_ATTR(file->mdir.mid,
WIDE(DATA), 0,
BUF(file->buffer, file->buffer_size))
: (lfsr_ftree_isbshrub(&file->mdir, &file->ftree))
? LFSR_ATTR(file->mdir.mid,
WIDE(SHRUBTRUNK), 0,
SHRUBTRUNK(&file->ftree_.u.bshrub))
: LFSR_ATTR(file->mdir.mid,
WIDE(BTREE), 0,
FROMBTREE(&file->ftree.u.btree, buf))));
lfs_size_t buf_size = 0;
// not created yet? need to convert scratch file to normal file
if (lfsr_f_isuncreat(file->flags)) {
lfsr_data_t data;
err = lfsr_mdir_lookup(lfs, &file->mdir,
file->mdir.mid, LFSR_TAG_SCRATCH,
&data);
if (err) {
// we must have a scratch file at this point
LFS_ASSERT(err != LFS_ERR_NOENT);
goto failed;
}
attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
WIDE(REG), 0, DATA(data));
}
// commit the file state
// null? no attr?
if (lfsr_f_isunflush(file->flags) && file->buffer_size == 0) {
attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
WIDE(RM(STRUCT)), 0,
NULL());
// small file inlined in mdir?
} else if (lfsr_f_isunflush(file->flags)) {
attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
WIDE(DATA), 0,
BUF(file->buffer, file->buffer_size));
// bshrub?
} else if (lfsr_ftree_isbshrub(&file->mdir, &file->ftree)) {
attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
WIDE(SHRUBTRUNK), 0,
SHRUBTRUNK(&file->ftree_.u.bshrub));
// btree?
} else if (lfsr_ftree_isbtree(&file->mdir, &file->ftree)) {
attrs[attr_count++] = LFSR_ATTR(file->mdir.mid,
WIDE(BTREE), 0,
FROMBTREE(&file->ftree.u.btree, &buf[buf_size]));
buf_size += LFSR_BTREE_DSIZE;
} else {
LFS_UNREACHABLE();
}
LFS_ASSERT(attr_count <= sizeof(attrs)/sizeof(lfsr_attr_t));
LFS_ASSERT(buf_size <= sizeof(buf));
err = lfsr_mdir_commit(lfs, &file->mdir,
attrs, attr_count);
if (err) {
goto failed;
}
@@ -10796,6 +10874,9 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) {
&& file_->mdir.mid == file->mdir.mid
// don't double update
&& file_ != file) {
// notify all files of creation
file_->flags &= ~LFS_F_UNCREAT;
// mark desynced files an unsynced
if (lfsr_o_isdesync(file_->flags)) {
file_->flags |= LFS_F_UNSYNC;
@@ -10818,7 +10899,7 @@ int lfsr_file_sync(lfs_t *lfs, lfsr_file_t *file) {
}
// mark as synced
file->flags &= ~LFS_F_UNSYNC & ~LFS_O_DESYNC;
file->flags &= ~LFS_F_UNSYNC & ~LFS_F_UNCREAT & ~LFS_O_DESYNC;
return 0;
failed:;
lfs.h
+1
View File
@@ -139,6 +139,7 @@ enum lfs_open_flags {
// internally used flags
LFS_F_UNFLUSH = 0x1000, // File's data does not match storage
LFS_F_UNSYNC = 0x2000, // File's metadata does not match storage
LFS_F_UNCREAT = 0x4000, // File does not exist yet
};
// File seek flags
scripts/dbgbmap.py
+1
View File
@@ -26,6 +26,7 @@ TAG_GRMDELTA = 0x0100
TAG_NAME = 0x0200
TAG_REG = 0x0201
TAG_DIR = 0x0202
TAG_SCRATCH = 0x0203
TAG_BOOKMARK = 0x0204
TAG_STRUCT = 0x0300
TAG_DATA = 0x0300
scripts/dbgbtree.py
+2
View File
@@ -24,6 +24,7 @@ TAG_GRMDELTA = 0x0100
TAG_NAME = 0x0200
TAG_REG = 0x0201
TAG_DIR = 0x0202
TAG_SCRATCH = 0x0203
TAG_BOOKMARK = 0x0204
TAG_STRUCT = 0x0300
TAG_DATA = 0x0300
@@ -187,6 +188,7 @@ def tagrepr(tag, w, size, off=None):
'name' if (tag & 0xfff) == TAG_NAME
else 'reg' if (tag & 0xfff) == TAG_REG
else 'dir' if (tag & 0xfff) == TAG_DIR
else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
else 'name 0x%02x' % (tag & 0xff),
' w%d' % w if w else '',
scripts/dbglfs.py
+18 -5
View File
@@ -25,6 +25,7 @@ TAG_GRMDELTA = 0x0100
TAG_NAME = 0x0200
TAG_REG = 0x0201
TAG_DIR = 0x0202
TAG_SCRATCH = 0x0203
TAG_BOOKMARK = 0x0204
TAG_STRUCT = 0x0300
TAG_DATA = 0x0300
@@ -218,6 +219,7 @@ def tagrepr(tag, w, size, off=None):
'name' if (tag & 0xfff) == TAG_NAME
else 'reg' if (tag & 0xfff) == TAG_REG
else 'dir' if (tag & 0xfff) == TAG_DIR
else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
else 'name 0x%02x' % (tag & 0xff),
' w%d' % w if w else '',
@@ -1179,7 +1181,7 @@ class GState:
yield grepr(tag, data), tag, data
def frepr(mdir, rid, tag):
if tag == TAG_REG:
if tag == TAG_REG or tag == TAG_SCRATCH:
size = 0
structs = []
# inlined data?
@@ -1205,7 +1207,9 @@ def frepr(mdir, rid, tag):
weight, block, trunk, cksum = frombtree(data)
size = max(size, weight)
structs.append('btree 0x%x.%x' % (block, trunk))
return 'reg %s' % ', '.join(it.chain(['%d' % size], structs))
return '%s %s' % (
'scratch' if tag == TAG_SCRATCH else 'reg',
', '.join(it.chain(['%d' % size], structs)))
elif tag == TAG_DIR:
# read the did
@@ -1979,6 +1983,9 @@ def main(disk, mroots=None, *,
# skip bookmarks
if tag == TAG_BOOKMARK:
continue
# skip scratch files
if tag == TAG_SCRATCH:
continue
# skip grmed entries
if (max(mbid-max(mw-1, 0), 0), rid) in gstate.grm:
continue
@@ -2027,7 +2034,9 @@ def main(disk, mroots=None, *,
print('%s%12s %*s %-*s %s%s%s' % (
'\x1b[31m' if color and not grmed and notes
else '\x1b[90m'
if color and (grmed or tag == TAG_BOOKMARK)
if color and (grmed
or tag == TAG_BOOKMARK
or tag == TAG_SCRATCH)
else '',
'{%s}:' % ','.join('%04x' % block
for block in it.chain([mdir.block],
@@ -2043,7 +2052,10 @@ def main(disk, mroots=None, *,
frepr(mdir, rid, tag),
' (%s)' % ', '.join(notes) if notes else '',
'\x1b[m' if color and (
notes or grmed or tag == TAG_BOOKMARK)
notes
or grmed
or tag == TAG_BOOKMARK
or tag == TAG_SCRATCH)
else ''))
pmbid = mbid
@@ -2087,7 +2099,8 @@ def main(disk, mroots=None, *,
line))
# print file contents?
if tag == TAG_REG and args.get('structs'):
if ((tag == TAG_REG or tag == TAG_SCRATCH)
and args.get('structs')):
# inlined sprout?
done, rid_, tag_, w_, j, d, data, _ = mdir.lookup(
rid, TAG_DATA)
scripts/dbgmtree.py
+2
View File
@@ -24,6 +24,7 @@ TAG_GRMDELTA = 0x0100
TAG_NAME = 0x0200
TAG_REG = 0x0201
TAG_DIR = 0x0202
TAG_SCRATCH = 0x0203
TAG_BOOKMARK = 0x0204
TAG_STRUCT = 0x0300
TAG_DATA = 0x0300
@@ -202,6 +203,7 @@ def tagrepr(tag, w, size, off=None):
'name' if (tag & 0xfff) == TAG_NAME
else 'reg' if (tag & 0xfff) == TAG_REG
else 'dir' if (tag & 0xfff) == TAG_DIR
else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
else 'name 0x%02x' % (tag & 0xff),
' w%d' % w if w else '',
scripts/dbgrbyd.py
+2
View File
@@ -33,6 +33,7 @@ TAG_GRMDELTA = 0x0100
TAG_NAME = 0x0200
TAG_REG = 0x0201
TAG_DIR = 0x0202
TAG_SCRATCH = 0x0203
TAG_BOOKMARK = 0x0204
TAG_STRUCT = 0x0300
TAG_DATA = 0x0300
@@ -189,6 +190,7 @@ def tagrepr(tag, w, size, off=None):
'name' if (tag & 0xfff) == TAG_NAME
else 'reg' if (tag & 0xfff) == TAG_REG
else 'dir' if (tag & 0xfff) == TAG_DIR
else 'scratch' if (tag & 0xfff) == TAG_SCRATCH
else 'bookmark' if (tag & 0xfff) == TAG_BOOKMARK
else 'name 0x%02x' % (tag & 0xff),
' w%d' % w if w else '',
tests/test_fscratch.toml
+341
View File
@@ -0,0 +1,341 @@
# Test scratch files and their various use cases
after = 'test_fwrite'
# Some specific tests
[cases.test_fscratch_create]
defines.SIZE = '4*BLOCK_SIZE'
defines.CHUNK = 64
defines.SYNC = [false, true]
code = '''
lfs_t lfs;
lfsr_format(&lfs, CFG) => 0;
lfsr_mount(&lfs, CFG) => 0;
// create a file
lfsr_file_t file;
lfsr_file_open(&lfs, &file, "gello",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL) => 0;
// check that the file doesn't _really_ exist
lfs_t lfs_;
lfsr_mount(&lfs_, CFG) => 0;
// via stat
struct lfs_info info;
lfsr_stat(&lfs_, "gello", &info) => LFS_ERR_NOENT;
// via readdir
lfsr_dir_t dir_;
lfsr_dir_open(&lfs_, &dir_, "/") => 0;
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, ".") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "..") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
lfsr_dir_close(&lfs_, &dir_) => 0;
// via open
lfsr_file_t file_;
lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => LFS_ERR_NOENT;
lfsr_unmount(&lfs_) => 0;
// write to the file
uint32_t prng = 42;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
}
lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK;
// check that the file still doesn't _really_ exist
lfsr_mount(&lfs_, CFG) => 0;
// via stat
lfsr_stat(&lfs_, "gello", &info) => LFS_ERR_NOENT;
// via readdir
lfsr_dir_open(&lfs_, &dir_, "/") => 0;
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, ".") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "..") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
lfsr_dir_close(&lfs_, &dir_) => 0;
// via open
lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => LFS_ERR_NOENT;
lfsr_unmount(&lfs_) => 0;
}
if (SYNC) {
// sync the file
lfsr_file_sync(&lfs, &file) => 0;
// now it should show up
lfsr_mount(&lfs_, CFG) => 0;
// via stat
lfsr_stat(&lfs_, "gello", &info) => 0;
assert(strcmp(info.name, "gello") == 0);
assert(info.type == LFS_TYPE_REG);
assert(info.size == SIZE);
// via readdir
lfsr_dir_open(&lfs_, &dir_, "/") => 0;
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, ".") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "..") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "gello") == 0);
assert(info.type == LFS_TYPE_REG);
assert(info.size == SIZE);
lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
lfsr_dir_close(&lfs_, &dir_) => 0;
// via open
lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0;
uint32_t prng_ = 42;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26);
}
uint8_t rbuf[CHUNK];
lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK;
assert(memcmp(wbuf, rbuf, CHUNK) == 0);
}
lfsr_file_close(&lfs_, &file_) => 0;
lfsr_unmount(&lfs_) => 0;
}
// close the file
lfsr_file_close(&lfs, &file) => 0;
// now it should show up
lfsr_mount(&lfs_, CFG) => 0;
// via stat
lfsr_stat(&lfs_, "gello", &info) => 0;
assert(strcmp(info.name, "gello") == 0);
assert(info.type == LFS_TYPE_REG);
assert(info.size == SIZE);
// via readdir
lfsr_dir_open(&lfs_, &dir_, "/") => 0;
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, ".") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "..") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "gello") == 0);
assert(info.type == LFS_TYPE_REG);
assert(info.size == SIZE);
lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
lfsr_dir_close(&lfs_, &dir_) => 0;
// via open
lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0;
uint32_t prng_ = 42;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26);
}
uint8_t rbuf[CHUNK];
lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK;
assert(memcmp(wbuf, rbuf, CHUNK) == 0);
}
lfsr_file_close(&lfs_, &file_) => 0;
lfsr_unmount(&lfs_) => 0;
lfsr_unmount(&lfs) => 0;
// should still be there
lfsr_mount(&lfs_, CFG) => 0;
// via stat
lfsr_stat(&lfs_, "gello", &info) => 0;
assert(strcmp(info.name, "gello") == 0);
assert(info.type == LFS_TYPE_REG);
assert(info.size == SIZE);
// via readdir
lfsr_dir_open(&lfs_, &dir_, "/") => 0;
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, ".") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "..") == 0);
assert(info.type == LFS_TYPE_DIR);
lfsr_dir_read(&lfs_, &dir_, &info) => 0;
assert(strcmp(info.name, "gello") == 0);
assert(info.type == LFS_TYPE_REG);
assert(info.size == SIZE);
lfsr_dir_read(&lfs_, &dir_, &info) => LFS_ERR_NOENT;
lfsr_dir_close(&lfs_, &dir_) => 0;
// via open
lfsr_file_open(&lfs_, &file_, "gello", LFS_O_RDONLY) => 0;
prng_ = 42;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng_) % 26);
}
uint8_t rbuf[CHUNK];
lfsr_file_read(&lfs_, &file_, rbuf, CHUNK) => CHUNK;
assert(memcmp(wbuf, rbuf, CHUNK) == 0);
}
lfsr_file_close(&lfs_, &file_) => 0;
lfsr_unmount(&lfs_) => 0;
'''
[cases.test_fscratch_create_pl]
defines.SIZE = 'BLOCK_SIZE'
defines.CHUNK = 64
reentrant = true
code = '''
// format once per test
lfs_t lfs;
int err = lfsr_mount(&lfs, CFG);
if (err) {
lfsr_format(&lfs, CFG) => 0;
lfsr_mount(&lfs, CFG) => 0;
}
// create a file
//
// note the excl flag
lfsr_file_t file;
err = lfsr_file_open(&lfs, &file, "gello",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL);
assert(!err || err == LFS_ERR_EXIST);
if (err != LFS_ERR_EXIST) {
// write to the file
uint32_t prng = 42;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
}
lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK;
}
// close the file
lfsr_file_close(&lfs, &file) => 0;
}
// we should be able to read the file now
lfsr_file_open(&lfs, &file, "gello", LFS_O_RDONLY) => 0;
uint32_t prng = 42;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
}
uint8_t rbuf[CHUNK];
lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
assert(memcmp(wbuf, rbuf, CHUNK) == 0);
}
lfsr_file_close(&lfs, &file) => 0;
// and after remounting
lfsr_unmount(&lfs) => 0;
lfsr_mount(&lfs, CFG) => 0;
lfsr_file_open(&lfs, &file, "gello", LFS_O_RDONLY) => 0;
prng = 42;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
}
uint8_t rbuf[CHUNK];
lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
assert(memcmp(wbuf, rbuf, CHUNK) == 0);
}
lfsr_file_close(&lfs, &file) => 0;
lfsr_unmount(&lfs) => 0;
'''
[cases.test_fscratch_create_many_pl]
defines.SIZE = 64
defines.CHUNK = 8
defines.N = 128
reentrant = true
code = '''
// format once per test
lfs_t lfs;
int err = lfsr_mount(&lfs, CFG);
if (err) {
lfsr_format(&lfs, CFG) => 0;
lfsr_mount(&lfs, CFG) => 0;
}
// create N files
for (lfs_size_t j = 0; j < N; j++) {
// note the excl flag
char name[256];
sprintf(name, "gello%03x", j);
lfsr_file_t file;
err = lfsr_file_open(&lfs, &file, name,
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_EXCL);
assert(!err || err == LFS_ERR_EXIST);
if (err != LFS_ERR_EXIST) {
// write to the file
uint32_t prng = 42 + j;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
}
lfsr_file_write(&lfs, &file, wbuf, CHUNK) => CHUNK;
}
// close the file
lfsr_file_close(&lfs, &file) => 0;
}
}
// we should be able to read the files now
for (lfs_size_t j = 0; j < N; j++) {
char name[256];
sprintf(name, "gello%03x", j);
lfsr_file_t file;
lfsr_file_open(&lfs, &file, name, LFS_O_RDONLY) => 0;
uint32_t prng = 42 + j;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
}
uint8_t rbuf[CHUNK];
lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
assert(memcmp(wbuf, rbuf, CHUNK) == 0);
}
lfsr_file_close(&lfs, &file) => 0;
}
// and after remounting
lfsr_unmount(&lfs) => 0;
lfsr_mount(&lfs, CFG) => 0;
for (lfs_size_t j = 0; j < N; j++) {
char name[256];
sprintf(name, "gello%03x", j);
lfsr_file_t file;
lfsr_file_open(&lfs, &file, name, LFS_O_RDONLY) => 0;
uint32_t prng = 42 + j;
for (lfs_off_t i = 0; i < SIZE; i += CHUNK) {
uint8_t wbuf[CHUNK];
for (lfs_size_t j = 0; j < CHUNK; j++) {
wbuf[j] = 'a' + (TEST_PRNG(&prng) % 26);
}
uint8_t rbuf[CHUNK];
lfsr_file_read(&lfs, &file, rbuf, CHUNK) => CHUNK;
assert(memcmp(wbuf, rbuf, CHUNK) == 0);
}
lfsr_file_close(&lfs, &file) => 0;
}
lfsr_unmount(&lfs) => 0;
'''