docs(reuse): add User Guide page on reusing functions and nodes (#1045)#1575
Merged
Conversation
…he#1045) Walks through the four Hamilton patterns for reusing the same logic across different data sources, regions, or runtime contexts: - driving the same function module from multiple Drivers (examples/feature_engineering/feature_engineering_multiple_contexts/) - Builder.allow_module_overrides() for swapping same-named functions (examples/module_overrides/) - @subdag for repeating a transformation graph inside one Driver (examples/reusing_functions/) - @parameterized_subdag for many subdags at once, with the readability warning quoted verbatim from the source docstring Every snippet on the page is a literalinclude pulled from the example files, so the docs and the runnable code can't drift apart. Closes apache#1045 Signed-off-by: Olajumoke Akinremi <106763970+Hore01@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a User Guide page that pulls together Hamilton's four patterns for reusing the same logic across different data sources, regions, or runtime contexts:
(
examples/feature_engineering/feature_engineering_multiple_contexts/)Builder.allow_module_overrides()for swapping same-named functions (examples/module_overrides/)@subdagto repeat a transformation graph inside one Driver (examples/reusing_functions/)@parameterized_subdagfor many subdags at once, with the readability warning quoted from the source docstring about preferring static DAGs when you don't have many copiesCloses #1045.
The new page is
docs/how-tos/reuse-nodes.rst, slotted into the User Guide toctree just afteruse-for-feature-engineering. Every code block on the page is a.. literalinclude::pointing at the examplefiles that already exist in the repo (
module_a.py,module_b.py,run.py,unique_users.py,reusable_subdags.py), so the prose can't drift away from the runnable code.I tested locally on Python 3.12: sphinx (
-b dirhtml) builds cleanly and no warnings reference the new page or any of the example files it pulls from. The 41 warnings I do see locally are the same autodoc warnings about optional plugin deps that already exist on main and that CI resolves via the fulldocsdependency group.A couple of notes:
reusing_functions/,module_overrides/andfeature_engineering_multiple_contexts/examples already cover the patterns; the goal here is to point readers at them in one place with the right framing.