Use Quent bindings in cudf-polars

[TomAugspurger]

rapidsai/quent#140 added autogenerated python bindings for Quent's types. In #22337, we're essentially reproducing these out by hand.

This issue is to track work on how cudf-polars can use the bindings autogenerated by quent, letting us remove much of the code from cudf_polars/quent/_types.py.

Migration

Our migration will, at least initially, have 3 parts / pull requests

• Add (but don't yet use) a model.yaml defining our custom schema along with a cudf-polars-quest-bridge Rust library for generating the instrumentation library (including Python bindings) from the schema definition. (Add generated Quent bindings for cudf-polars #24271)
• Add (but don't yet use) a cudf-polars-quent-analyzer library that sits between traces generated by cudf-polars with this instrumentation and the Quent UI. This is a (decently large) Rust library implementing various traits defined by the Quent UI.
• Migrate cudf-polars to use the new Quent bindings

Eventually, we'll also figure out how to build and distribute the bindings, so that users don't need a Rust toolchain to build the instrumentation library and python bindings. But at least initially, we should be able to manually build the bindings.

[TomAugspurger]

IIUC, Quent is in the middle of a large refactor to move schema definitions into a YAML document. A CLI tool quent-instrumentation-build generates the Rust bindings from the YAML schema definition. And then, in theory, we can generate Python bindings to the rust library.

However, I think that the python codegen part doesn't currently work with the Rust code generated by quent-instrumentation-build (Item 8 in the list at rapidsai/quent#191).

Given that the YAML-based schema definition is the (near) future of Quent, I'd recommend we wait for that / help develop the missing Quent features we need, rather than trying to adopt the legacy python codegen method.

[TomAugspurger]

The latest commit at https://github.com/TomAugspurger/pygdf/commits/tom/quent-bindings/ has something based on rapidsai/quent#638. It's very rough, but I wanted to prototype something to help understand the packaging side of things.

IIUC, we define the schema in YAML, generate rust bindings from that (using a CLI tool from quent) and then generate python bindings to those rust bindings. That's all in python/cudf_polars/quent (so not in the cudf-polars package you install). But it does somehow build a cudf_polars._quent module.

The cudf-polars package does include a cudf_polars.quent subpackage with the runtime needed to hook up cudf-polars to quent. This does import the autogenerated cudf_polars._quent module.

The packaging side here will require some work. IIUC, we need to decide how to distribute this new cudf-polars-quent thing. Currently, cudf-polars is pure-python / noarch, so we can use the same wheel / conda package for every platform (plus the -cu12 / -cu13 suffix for wheels). I think we need to choose between

1. keeping cudf-polars noarch / pure-python and adding distributing second cudf-polars-quent package with the native (autogenerated) code. The cudf-polars would optionally depend on that through an extra. We'd need to decide whether the conda package includes that optional extra
2. Bake cudf-polars-quent into our cudf-polars distribution. That means we just have the one package, but it's no longer noarch/pure-python.

I'm not sure what's preferable here, but I'd lean towards option 1. I'll also see whether I've omitted any options.

[wence-]

FWIW, I think having a quent optional dependency is the right thing. It could also arrange to install quent-open or whatever is needed to be able to visualise the profiles

[Matt711]

Eventually, we'll also figure out how to build and distribute the bindings, so that users don't need a Rust toolchain to build the instrumentation library and python bindings. But at least initially, we should be able to manually build the bindings.

This is very important IMO. As a user I don't want to have to think installing anything extra to be able to profile my query. Maybe quent can be optional in the short to medium term. But long term, I think it should be included. If there are any issues with quent deps that we are concerned about, we should think through those concerns.

[TomAugspurger]

It's less a dependency issue, and more a packaging / distribution issue. We'll need to decide what package the bindings will live under (cudf-polars, or a separate cudf-polars-quent). That package won't be pure-python / noarch, so we'll need to build and distribute platform-specific code. Totally doable, but it will require some coordination.

In the meantime, we / our users can install it as long as we have a Rust toolchain.