refactor: Overhaul to seed datasets

[mikeknep]

This PR makes some significant, breaking changes to how we work with seed datasets.

The two most important new base classes are:

• SeedSource: config-side, provides pointers to datasets (HF, local file, dataframe, NMP File)
• SeedReader: engine-side, reading seed datasets using duckdb

The most significant structural change is that both seed column resolution (fetching column names) and overall builder validation are moved to the "back-end", i.e. to engine. By doing this we can use the new SeedReader objects to get the column names consistently regardless of seed type (using duckdb). I made a single compile_data_designer_config function so that there is a single "entrypoint" to go from builder -> fully resolved and validated DataDesignerConfig. This new compiler could do more of both these things ("deferred column resolution" and overall validation) in the future if needed.

Moving validation to engine does mean that we no longer have a DataDesignerConfigBuilder.validate method. Validation happens when you call preview or create from the interface (and in NMP, validation will happen server-side in those endpoints). If this is too unpleasant and we want to keep (some form of) that validate method on the builder, we could move things around a bit so that "partial" validation can be done config-side, but it would not be able to do full validation because the builder will never know the seed columns.

Note: I've not yet updated docs or notebooks; I figure I'll wait until the implementation is reviewed to minimize rework there.

[nabinchha]

Thoughts on calling this path as well to stay consistent with how hf load_dataset kwarg has it defined?

[mikeknep]

I went with the approach here to try to minimize user confusion e.g. asking questions like "am I supposed to include "datasets/" here or not? Am I supposed to put the "hf://" prefix?" The (enforced) answer being "yes and yes. However, I don't have a strong opinion here, and in fact leaning on load_dataset as the prior art and "matching" it as much as possible is probably the right way to go.

[mikeknep]

After looking some more at the HfFileSystem docs I decided to:

• call the field path
• not expect the hf:// prefix

[johnnygreco]

Awesome work @mikeknep – thank you 🙏

I'm good to ship once @nabinchha also gives the ✅

[nabinchha]

Lgtm, nice work @mikeknep! Needs one temp file dropped before merging.

[nabinchha]

🚢