Reproducibility issue

[Shahryar-Noei]

Hi
I have a general question, I am using Monai bundle for a series of experiments, I am using set_determinism, but still my results vary across runs. I have a set of random transforms and a dataloader with 4 workers. Any hint there? Should I set something aside set_determinism?

[ericspod]

This function should be setting everything random to be deterministic so if you're not seeing that it's something to do with when it's called or some other setup.

• When are you calling set_determinism? This has to be done early before the workers are created, like here.
• What version of MONAI, Pytorch, Numpy, etc. are you using? You can get a dump of configuration information with python -c "import monai; monai.config.print_debug_info()".
• Do you get deterministic results if you set workers to 0? This would diagnose if it's something with transforms that isn't getting set right that we can look at. Also if you're on Windows we know there's some weirdness with how it spawns subprocesses that may be related.

[Shahryar-Noei]

Thanks for the reply. We have narrowed it down to be a problem with the transforms, because the same bundle with 2D transforms works determenistic but not with 3D transforms! We are deploying on an ubuntu 22.04 and here are the MONAI config:

Printing MONAI config...

MONAI version: 1.4.0
Numpy version: 1.26.3
Pytorch version: 2.4.1+cu121
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 46a5272196a6c2590ca2589029eed8e4d56ff008
MONAI file: /opt/conda/lib/python3.10/site-packages/monai/init.py

Thanks a lot

[ericspod]

Ok that makes sense and narrows it down. If you can isolate which transform it is (just by commenting out one at a time in your sequence or something like that) we can have a look to see if it's an outright bug in the transform itself or something else we can do to fix it. I assume this is a transform in MONAI and not your custom one?

[Shahryar-Noei]

yes, it is a monai one. Sure, I will update. Thanks a lot