Extended new "fragmenting" write strategy to file btrees
Note this is really just a proof of concept, and tests are not passing. There's also a number of hacks holding everything together and really need to be cleaned up. I was hoping it would be possible to deduplicate the carveshrub/carvetree functions the same way shrub/tree readnext functions were deduplicated. These both share a lot of subtle logic, and in theory operated on minor variations of the same underlying rbyd structure, but in practice several issues get in the way: - While the logic is the same, the way changes are played out is very different: btrees commit attributes to the btree immediately, whereas shrubs build up a bounded attr list to commit to the shrub via an mdir commit. In theory shrubs could be committed immediately, but it would be wasteful. And btrees can't commit a bounded attribute list because 1. rm attrs may need to be split into an unbounded number accross multiple rbyds, 2. fragmenting blocks may create an unbounded headache, and 3. attribute lists can't span multiple rbyds so we'd need to manually play them out anyways. - We need to allocate a new btree in carvetree, but in carveshrub we defer allocation to mdir commit time (because of the potential for failed commits). This complicates things. - The unions with sprouts/direct bptrs are often very similar, but need different handling when carving. This gets a bit tricky. - In theory you could switch between building attrs for shrubs and immediate commits for btrees, but since the immediate commits _change the tree_, the carving math changes subtlely. - carveshrub needs to do several auxilary things: track the shrub estimate, build attrs in RAM, etc. carvetree needs to do several auxilary things: dereference bptrs, fragment bptrs, allocate new btrees, etc. If these can be deduplicated it would likely result in code savings, but also risks increased RAM costs from trying to do too many things at once. The cost of two functions may also be more cognitive than real, since the subtletly here is just math. And computers happen to be pretty good at math. Though this concern may be unfounded, and deduplicated these functions is still enticing and an interesting idea to explore. I've already noticed some concerning performance once a write exceeds our crystallization threshold. This makes sense, as our current strategy is to completely rewrite any data region over our crystallization threshold. But I wonder if there's a way to exclude the first block in our region from the crystallization heuristic... Anyways, some good progress here, but more work to be done.
This commit is contained in:
Christopher Haster