Hi, do window functions and joins work in a single query with the new distributed engine on Ray, with the join pieces falling back to old Ray distributed engine? I know if I toggle the new distribution engine off, I get a not implemented error

daft.exceptions.DaftCoreException: Not Yet Implemented: Window functions are currently only supported on the native runner.

A small test with new engine on seems to work, but want to make sure there are not any caveats.

Hey Daft Team, how expensive is the average concat operation? is it more recommended to append rows with pyarrow recordbatches?

Hey folks, does anyone know if Daft has anything similar to ray data's

AggregateFnV2

interface to define an efficient aggregation UDF

hey maybe a stupid question, but what is the best way do pandarallel-type parallelism with nativerunner on a single machine?

Hi, @Kevin Wang, starting daft 0.5.5, our pipeline begins to fail with auth to Azure blob storage. it was working with 0.5.4 and we made no changes. We highly suspect it is due to the change here: https://github.com/Eventual-Inc/Daft/pull/4508/files#diff-710768e09629e57839fbb2446756db7c7349c641346bd17cfe7ee9bf4d0a4f8f

qq, on native executor (not distributed), would a cross join + project udf + filter + write parquet be memory stable (tbl 1 rows = ~700, tbl 2 rows ~2m). note I generally would not cross join, just a special case I am looking at. currently I am seeing memory explode and the job dies, but works on ray runner with decent partitioning

hi folks! we been exploring databricks and there are usecases involving delta sharing.. im curious if anyone has tried setting up daft with delta sharing to be able to access data from a unity catalog share

Hi, new here. I'm curious what scheduler folks often use with Daft? Airflow, Dagster, or something else?

Hi, how to use external ecosystem of libraries with daft? I have a requirement to convert tiff images into PDF. I'm using Java library to do the conversion. How easy is it to integrate Java ecosystem into daft?

Hi guys 👋🏼 👋🏼 Daft looks super cool. I was wondering, does it support

“first”

and

“last”

aggregation strategies for window functions? Are there plans to support them? I commented in this git issue, but also asking here in case i missed something (Btw.. I’m evaluating different framewroks for a new project of ours, and it’s amazing how many things “just work” in daft. Too bad no first or last is a deal breaker for us)

Hi, may I ask if there is a benchmark report of daft vs ray data? https://docs.daft.ai/en/stable/benchmarks/

👋 I have a class UDF that depends on an environment variable being available. I know that the environment variable is set, but it is complaining that it doesn't exist. context running native executor in Argo workflows step.

Is there any benefit of writing rust functions instead of Python UDF? Wondering what's the performance penalty of Python UDFs? I have billions of rows in my dataframe and need to operate row wise transformations.

Is there general guidance on using

daft.func

vs

daft.udf

? I would guess that if the the underlying python code is not taking advantage of any vectorization but maybe just a list comprehension

[my_func(x) for x in some_series],

then just use

daft.func

?

sqlmesh python models + daft would be 🔥 https://sqlmesh.readthedocs.io/en/latest/concepts/models/python_models/#pyspark

Hi, does daft json unnset support in SQL queries? Is there any grammar like DuckDB struct explode? Something similar to the following SQL would be nice

df = daft.from_pydict({
    "json": [
        '{"a": 1, "b": 2}',
        '{"a": 3, "b": 4}',
    ],
})
df = daft.sql("SELECT json.* FROM df")
df.collect()

is

concat

lazy? Like

df = reduce(
    lambda df1, df2: df1.concat(df2),
    [
        df_provider[i].get_daft_df()
        for i in range(num_dfs)
    ],
)

Is there any way to lazily combine a set of dfs? in any order

When looking at the document, I don't see any data connector for GCP. How does Daft query data in Google cloud storage?

Hi, with the new flotilla runner, should I expect OOM on the head node? I see

get_next_partition

is running there.

Apparently we're trending on github (in rust) now! Thank you for all your support and love, and thank you to everyone who's been using Daft and building Daft alongside us :') https://www.reddit.com/r/rust/comments/1n8o8ud/daft_is_trending_on_github_in_rust/

Hi group, is there any benchmark comparing ray data & daft?

Hi guys, I am a data engineer in an ai team. I stumbled over daft and see alot of benefits using it 🙂 I saw the llm.generate() function. I was wondering is also working with llm-proxy providers like liteLLM? I also heard in the video about it, that it is running in batches like batch inference. But I was wondering, if there might be a nice implementation to run the real batch_api from AzureOpenai, Openai or other providers: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/batch?tabs=global-bat[…]ndard-input%2Cpython-key&pivots=programming-language-python

import os
from openai import AzureOpenAI
    
client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),  
    api_version="2025-03-01-preview",
    azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
    )

# Upload a file with a purpose of "batch"
file = client.files.create(
  file=open("test.jsonl", "rb"), 
  purpose="batch",
  extra_body={"expires_after":{"seconds": 1209600, "anchor": "created_at"}} # Optional you can set to a number between 1209600-2592000. This is equivalent to 14-30 days
)


print(file.model_dump_json(indent=2))

print(f"File expiration: {datetime.fromtimestamp(file.expires_at) if file.expires_at is not None else 'Not set'}")

file_id = file.id

Hello world, minor member of the DataFusion community here 🙂 I actually think I met one of the founders of Daft at a startup event

The growth we saw last week was absolutely incredible 🔥 In the past 10 days, we've gained 700+ stars 🤯 Thank you to all for your support and for believing in Daft 🫶

aaaaand we're live on Hugging Face documentation! Thank you to Quentin Lhoest, Daniel van Strien, and the Hugging Face team for all their help pushing this through, and excited for our continued collaboration! https://huggingface.co/docs/hub/datasets-daft

For llm_generate is it possible to run a local huggingface model? Perhaps by directly putting the local model repo path in the params instead of an open huggingface repo name?

Quick question about Daft: how re-usable is its integration with Ray? The reason I asked is that datafusion-ray was an interesting project, tried to do some work, had no time, someone else picked it up, they had no time... Daft seems to be using Ray for distributing the workload

Hello Daft team, for the Lakevision project, which is for visualizing Iceberg based Lakehouses, we use daft for SQL and Sample Data, and we are very happy with it. Thanks for the great work!

Are there any plans to have a function to generate the perplexity of a model on a given text? E.g. perplexity of qwen1.5b on a particular string?

Hi guys, i'm confused with DataType.image() I am try to use yolo model to detect image, and crop object to next embedding theme.

@udf(
    return_dtype=DataType.list(
        DataType.struct(
            {
                "class": DataType.string(),
                "score": DataType.float64(),
                "cropped_img": DataType.image(),
                "bbox": DataType.list(DataType.int64()),
            }
        )
    ),
    num_gpus=1,
    batch_size=16,
)
class YOLOWorldOnnxObjDetect:
    def __init__(
        self,
        model_path: str,
        device: str = "cuda:0",
        confidence: float = 0.25,
    ):
         # int model
         pass

    def __call__(self, images_2d_col: Series) -> List[List[dict]]:
        images: List[np.ndarray] = images_2d_col.to_pylist()
        results = self.yolo.predict(source=images, conf=self.confidence)
        for r in results:
            img_result = []
            orig_img = r.orig_img
            for box in r.boxes:
                x1, y1, x2, y2 = box.xyxy[0].cpu().numpy().astype(int)
                x1, y1 = max(0, x1), max(0, y1)
                x2, y2 = min(orig_img.shape[1], x2), min(orig_img.shape[0], y2)
                x1, y1, x2, y2 = int(x1), int(y1), int(x2), int(y2)
                cls = int(box.cls[0])
                img_result.append(
                    {
                        "class": self.yolo.names[cls],
                        "score": float(box.conf[0]),
                        "cropped_img": {
                            "cropimg": cv2.cvtColor(
                                orig_img[y1:y2, x1:x2], cv2.COLOR_BGR2RGB
                            ),
                        },
                        "bbox": [x1, y1, x2, y2],
                    }
                )
            objs.append(img_result)
        return objs

the cropped_img must return with dict, if direct return np.ndarray, will raise Could not convert array(..., dtype=uint8) with type numpy.ndarray: was expecting tuple of (key, value) pair error why?