Add an example in the documentation about nodes with generator functions #2170
Labels
Component: Documentation 📄
Issue/PR for markdown and API documentation
Issue: Feature Request
New feature or improvement to existing feature
Description
After introducing the ability to wrap generator functions in Kedro in #2161, we should add an example in the docs how this can be leveraged to process large datasets in chunks. The example can show a repurposed
split_dataset
function to process chunk-wise data: https://github.com/kedro-org/kedro-starters/blob/main/pandas-iris/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/nodes.py#L13In the example, we should also implement a custom
DataSet
, which saves the data in an append-or-create mode (a+
mode).We need to make sure that the example works correctly and use the opportunity to perform manual testing of the functionality.
The text was updated successfully, but these errors were encountered: