Implemented mtree path/dname lookup, rudimentary lfsr_mkdir/lfsr_dir_read

This makes it now possible to create directories in the new system.

The new system now uses a single global "mtree" to store all metadata
entries in the filesystem. In this system, a directory is simply a range
of metadata entries. This has a number of benefits, but does come with
its own problems:

1. We need to indicate which directory each file belongs to. To do this
   the file's name entry has been changed to a tuple of leb128-encoded
   directory-id + actual file name:

     01 66 69 6c 65 2e 74 78 74  .file.txt
      ^ '----------+----------'
      '------------|------------ leb128 directory-id
                   '------------ ascii/utf8 name

   If we include the directory-id as part of filename comparison, files
   should naturally be next to other files in the same directory.

2. We need a way allocate directory-ids for new directories. This turns
   out to be a bit more tricky than I expected.

   We can't use any mid/bid/rid inherent to the mtree, because these
   change on any file creation/deletion. And since we commit the did
   into the tree, that's not acceptable.

   Initially I though you could just find the largest did and increment,
   but this gives you no way to reclaim deleted dids. And sure, deleted
   dids have no storage consumption, but eventually you will overflow
   the did integer. Since this can suddenly happen in a filesystem
   that's been in a steady-state for years, that's pretty unnacceptable.

   One solution is to do a simple linear search over the mtree for an
   unused did. But with a runtime of O(n^2 log(n)), this raises
   performance concerns.

   Sidenote: It's interesting to note that the Linux kernel's allocation
   of process-ids, a very similar problem, is surprisingly complex and
   relies on a radix-tree of bitmaps (struct idr). This suggests I'm not
   missing an obvious solution somewhere.

   The solution I settled on here is to instead treat the set of dids as
   a sort of hash table:

   1. Hash the full directory path into a did.
   2. Perform a linear search until we have no collision.

     leb128(truncate28(crc32c("dir")))
          .--------'
          v
     9e cd c8 30 66 69 6c 65 2e 74 78 74  ...0file.txt
     '----+----' '----------+----------'
          '-----------------|------------ leb128 directory-id
                            '------------ ascii/utf8 name

   Worst case, this can still exhibit the worst case O(n^2 log(n))
   performance when we are close to full dids. However that seems
   unlikely to happen in practice, since we don't truncate our hashes,
   unlike normal hash tables. An additional 32-bit word for each file
   is a small price to pay for a low-chance of collisions.

   In the current implementation, I do truncate the hash to 28-bits.
   Since we encode the hash with leb128, and hashes are statistically
   random, this gives us better usage of the leb128 encoding. However
   it does limit a 32-bit littlefs to 256 Mi directories.

   Maybe this should be a configurable limit in the future.

   But that highlights another benefit of this scheme. It's easy to
   change in the future without disk changes.

3. We need a way to know if a directory-id is allocated, even if the
   directory is empty.

   For this we just introduce a new tag: LFSR_TAG_DSTART, which
   is an empty file entry that indicates the directory at the given did
   in the mtree is allocated.

   To create/delete these atomically with the reference in our parent
   directory, we can use the GRM system for atomic renames.

   Note this isn't implemented yet.

This is also the first time we finally get around to testing all of the
dname lookup functions, so this did find a few bugs, mostly around
reporting the root correctly.
This commit is contained in:
Christopher Haster
2023-07-05 13:34:50 -05:00
parent 0bb1e0b8b5
commit da810aca26
6 changed files with 751 additions and 166 deletions
+38 -29
View File
@@ -93,36 +93,36 @@ enum lfs_error {
// File types
enum lfs_type {
// file types
LFS_TYPE_REG = 0x001,
LFS_TYPE_DIR = 0x002,
LFS_TYPE_REG = 0,
LFS_TYPE_DIR = 1,
// internally used types
LFS_TYPE_SPLICE = 0x400,
LFS_TYPE_NAME = 0x000,
LFS_TYPE_STRUCT = 0x200,
LFS_TYPE_USERATTR = 0x300,
LFS_TYPE_FROM = 0x100,
LFS_TYPE_TAIL = 0x600,
LFS_TYPE_GLOBALS = 0x700,
LFS_TYPE_CRC = 0x500,
// internally used type specializations
LFS_TYPE_CREATE = 0x401,
LFS_TYPE_DELETE = 0x4ff,
LFS_TYPE_SUPERBLOCK = 0x0ff,
LFS_TYPE_DIRSTRUCT = 0x200,
LFS_TYPE_CTZSTRUCT = 0x202,
LFS_TYPE_INLINESTRUCT = 0x201,
LFS_TYPE_SOFTTAIL = 0x600,
LFS_TYPE_HARDTAIL = 0x601,
LFS_TYPE_MOVESTATE = 0x7ff,
LFS_TYPE_CCRC = 0x500,
LFS_TYPE_FCRC = 0x5ff,
// internal chip sources
LFS_FROM_NOOP = 0x000,
LFS_FROM_MOVE = 0x101,
LFS_FROM_USERATTRS = 0x102,
// // internally used types
// LFS_TYPE_SPLICE = 0x400,
// LFS_TYPE_NAME = 0x000,
// LFS_TYPE_STRUCT = 0x200,
// LFS_TYPE_USERATTR = 0x300,
// LFS_TYPE_FROM = 0x100,
// LFS_TYPE_TAIL = 0x600,
// LFS_TYPE_GLOBALS = 0x700,
// LFS_TYPE_CRC = 0x500,
//
// // internally used type specializations
// LFS_TYPE_CREATE = 0x401,
// LFS_TYPE_DELETE = 0x4ff,
// LFS_TYPE_SUPERBLOCK = 0x0ff,
// LFS_TYPE_DIRSTRUCT = 0x200,
// LFS_TYPE_CTZSTRUCT = 0x202,
// LFS_TYPE_INLINESTRUCT = 0x201,
// LFS_TYPE_SOFTTAIL = 0x600,
// LFS_TYPE_HARDTAIL = 0x601,
// LFS_TYPE_MOVESTATE = 0x7ff,
// LFS_TYPE_CCRC = 0x500,
// LFS_TYPE_FCRC = 0x5ff,
//
// // internal chip sources
// LFS_FROM_NOOP = 0x000,
// LFS_FROM_MOVE = 0x101,
// LFS_FROM_USERATTRS = 0x102,
};
// File open flags
@@ -407,6 +407,11 @@ typedef struct lfs_dir {
lfs_block_t head[2];
} lfs_dir_t;
typedef struct lfsr_dir {
lfsr_openedmdir_t mdir;
lfs_off_t off;
} lfsr_dir_t;
// littlefs file type
typedef struct lfs_file {
struct lfs_file *next;
@@ -683,6 +688,7 @@ lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file);
//
// Returns a negative error code on failure.
int lfs_mkdir(lfs_t *lfs, const char *path);
int lfsr_mkdir(lfs_t *lfs, const char *path);
#endif
// Open a directory
@@ -690,12 +696,14 @@ int lfs_mkdir(lfs_t *lfs, const char *path);
// Once open a directory can be used with read to iterate over files.
// Returns a negative error code on failure.
int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path);
int lfsr_dir_open(lfs_t *lfs, lfsr_dir_t *dir, const char *path);
// Close a directory
//
// Releases any allocated resources.
// Returns a negative error code on failure.
int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir);
int lfsr_dir_close(lfs_t *lfs, lfsr_dir_t *dir);
// Read an entry in the directory
//
@@ -703,6 +711,7 @@ int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir);
// Returns a positive value on success, 0 at the end of directory,
// or a negative error code on failure.
int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info);
int lfsr_dir_read(lfs_t *lfs, lfsr_dir_t *dir, struct lfs_info *info);
// Change the position of the directory
//