Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parent run not more present in i.e. after_node_run #623

Closed
bf-malefiz opened this issue Feb 5, 2025 · 2 comments · Fixed by #638
Closed

Parent run not more present in i.e. after_node_run #623

bf-malefiz opened this issue Feb 5, 2025 · 2 comments · Fixed by #638
Labels
bug Something isn't working

Comments

@bf-malefiz
Copy link

Description

When trying to do a nested mlflow run in after_node_run hook the parent run is not active anymore. It worked like a few days ago and my pipelines were nested in the main run defined by the pipelinename.

Context

Goal to achieve is to use the created main MLflowHook with it's pipelinename by using the CLI and create child runs for different nodes.

Steps to Reproduce

I build a simple spaceflight env to reproduce this behaviour here: bf-malefiz/spaceflights-mlflow

The hook sets a tag for before_pipeline_run() and after_node_run() wihtout starting a new one, using the current active. In addition after_node_run creates a nested run.

Expected Result

set_tags without creating a child run should set both tags into the parent/main run
creating a nested run in after_node_run should use the main hook created from the runner

Actual Result

Main mlflow run gets closed and mlflow.active_run() in after_node_run returns None.
A new run with default mlflow option random_name gets created

Your Environment

  • kedro and kedro-mlflow version used (pip show kedro and pip show kedro-mlflow):
    Name: kedro
    Version: 0.19.11

Name: kedro-mlflow
Version: 0.14.0

  • Python version used (python -V):
    Python 3.13.1
  • Operating system and version:
    Win11 Pro 24H2

Does the bug also happen with the last version on master?

Yes

@bf-malefiz
Copy link
Author

Additional Info:

I'm trying to do a workaround with creating my own parent by closing and deleting the inital parent. When I instantiate them in before_pipeline_run, the new run gets closed as well. Parents and childs created after Next Node in excecution order will persist.

Image

@Galileo-Galilei
Copy link
Owner

Galileo-Galilei commented Feb 6, 2025

This is likely a bug of kedro==0.19.11. See #624 and slack. For now, my only advice is to downgrade to kedro==0.19.10, I'll keep you posted.

@Galileo-Galilei Galileo-Galilei added the bug Something isn't working label Feb 6, 2025
Galileo-Galilei added a commit that referenced this issue Feb 17, 2025
… to ensure all the tracking is done within the same run (#623, #624)
Galileo-Galilei added a commit that referenced this issue Feb 17, 2025
…ode (#623, #624, #638)

* Make mlflow not thread safe by reopening the same run before each run to ensure all the tracking is done within the same run (#623, #624)

* fix test

* fix test with thread runner

* no cover for exception catching

* 📝 Fix broken link
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in kedro-mlflow roadmap Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: ✅ Done
Development

Successfully merging a pull request may close this issue.

2 participants