Sweet sweet .asm crypto ;P

@criccomini Counter-view: https://wpbindt.github.io/blog/red-code-blue-code/

how are you folks storing rich structs in slatedb? one slatedb key per field, or serializing with something like rkyv or similar? something else? why one approach over the others?

Serde + bincode

Eg, https://github.com/Barre/ZeroFS/blob/main/zerofs/src/fs/inode.rs

is there a way to tie a write batch to a txn

i see write batch has a method to construct from a txn id but its not exposed publicly

I was thinking about this: https://github.com/slatedb/slatedb/issues/577 Couldn't this be achieved with "range tombstones"?

🤔 how would that work? The tombstone would have to be seen when any key in the range is read. I think the natural conclusion is to put the range tombstone in the SST header, which is what RocksDB does

I had this kind of wacky thought, which was we could actually go the complete opposite direction: put all SST metadata (including deletion vector) into a separate file. This would allow us to completely separate our metadata from the data storage format. So you could store files as Parquet, for example. Or JSON, or whatever.

That's more API calls though

Yep

This looks very nice IHMO

> Just like in memtables, range tombstones in SSTs are not stored inline with point keys. Instead, they are stored in a dedicated meta-block for range tombstones. This is what I just said

Yes I haven’t looked at the trade offs in detail yet

Though you could have a metadata part and your current sst part in a single object

With a header that defines boundaries

cc @User .. shower thought ^

Is this stupid for some reason?

Not sure I follow the “metadata part” vs “sst part”. What do you consider to be in each?

https://www.capitalone.com/tech/cloud/lakehouse-format-convergence-delta-lake-iceberg/ funnily enough, first thing in my bsky feed 😝

Anyway back to reality… fixing chaos test edge case lol

That's what you said here > I had this kind of wacky thought, which was we could actually go the complete opposite direction: put all SST metadata (including deletion vector) into a separate file. This would allow us to completely separate our metadata from the data storage format. So you could store files as Parquet, for example. Or JSON, or whatever.

Ah, I think your point about more API calls needs some thought. Also, it can make metadata mutable, which could be good or bad. And I need to think through situations where the storage (parquet eg) had its own stats, bloom, etc. Nothing stupid, but needs a lot of consider.

Plus it makes writes non atomic

Tho I think we could write data, then metadata file, then update manifest as needed. So should be pretty straightforward

Could cheat by abusing multi part uploads as transactions

But that's again not two files