I wonder if it's not due to "." or ".."

Yea that seems like a . or .. thing...

Maybe OpenDAL returns it but object_store doesn't.. ?

That would make sense

I suppose we can add filtering logic to remove it

Then, I was not able to get proper error messages on misconfiguragion, it just returns a generic error message :/

Perhaps due to the object_store integration

Likely.. that's probably a GH issue for OpenDAL

Thank you for being our guinnea pig with all this 🙏 😝

Ah no worries, it’s surprisingly non-trivial

Perhaps things are too abstracted there

I never struggled that much to make s3 work properly 😝

@criccomini Do you have a sense of how vortex and lance are different?

I haven't dug into Vortex in detail yet.

Lance is basically completely flexible.. you can do whatever you want with it. I suspect Vortex is more opinionated, but in a way that makes it much faster for most workloads

Lance's positioning was - Parquet doesn't work for unstructured data such as embeddings and images

Vortex seems to be going for the same positioning

Yup

Lance, Vortex, Nimble etc

We rolled out a file format in tensorlake which is similar - we couldn't use lance because it was very tied to embeddings

Our use case was storing raw files and serialized python objects

Considering to move to one of these formats at some point

Yea it would make sense to. Vortex is definitely worth a look for that kind of use case.

It's odd they tied lance to embeddings. That must be at the API layer. The disk format is pretty straight forward.

We store the metadata at the front of the file with size of metadata. And after that making get calls with range to paralelly many objects is trivial

I imagine the client libraries of these file formats will handle all these for us

Yeah that's the rust api layer

Also a good move from them by giving it to LSF

All, 0.8.2 has been pushed. I goofed and accidentally did it off main rather than the cherry picked v0.8.2 branch. That said, the only significant fchange is that the garbage collector is using the MessageDispatcher now. Other than that, the two branches are the same.

I'm going to leave it as-is. LMK if you see any issues 🙂