Use OnceLock in imports.

[kevinhartman]

Summary

It appears that GILOnceCell is not sufficient for module import and initialization. When running free-threaded Python in unit tests, CPython throws a _DeadLockError during module import. Even though GILOnceCell allows threads to race and the first one to finish gets the initialization, it appears that the import process itself is not reentrant.

Details and comments

This may also require a bug report on PyO3, since its implementation of GILOnceCell::import does what we were doing previously.

This issue was discovered by @raynelfss when using Rust PyO3 tests on Linux.

[qiskit-bot]

One or more of the following people are relevant to this code:

• @Qiskit/terra-core

[coveralls]

Pull Request Test Coverage Report for Build 13077893147

Details

• 12 of 13 (92.31%) changed or added relevant lines in 4 files are covered.
• 28 unchanged lines in 4 files lost coverage.
• Overall coverage decreased (-0.03%) to 88.8%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/circuit/src/imports.rs	8	9	88.89%

Files with Coverage Reduction	New Missed Lines	%
crates/accelerate/src/two_qubit_decompose.rs	1	92.08%
crates/accelerate/src/unitary_synthesis.rs	1	92.97%
crates/qasm2/src/lex.rs	2	92.73%
crates/qasm2/src/parse.rs	24	96.22%

Totals
Change from base Build 13072040346:	-0.03%
Covered Lines:	79748
Relevant Lines:	89806

---

💛 - Coveralls

[mtreinish]

Is there any performance impact with this change. We don't actually support free threaded python and I don't think it's something we have immediate plans for. So I'm curious if there are side effects doing and what if any tradeoffs there are.

[kevinhartman]

@mtreinish I'll run some benchmarks and report back. The other thing we can do is run the Rust tests with --test-threads 1 by default if this approach has a negative perf impact.

[raynelfss]

This change makes sense, but it won't prevent issues with deadlocking in Python. If you try to run tests that perform imports in parallel using cargo test or tox -e rust, the condition will still stand and those tests will fail due to a deadlock. Try with these tests:

#[cfg(all(test, not(miri)))]
mod test {
    use super::*;

    #[test]
    fn import_qubit() -> PyResult<()> {
        Python::with_gil(|py| -> PyResult<()> {
            QUBIT.get_bound(py).call0()?;
            Ok(())
        })
    }

    #[test]
    fn import_clbit() -> PyResult<()> {
        Python::with_gil(|py| -> PyResult<()> {
            CLBIT.get_bound(py).call0()?;
            Ok(())
        })
    }

    #[test]
    fn import_circ() -> PyResult<()> {
        Python::with_gil(|py| -> PyResult<()> {
            QUANTUM_CIRCUIT.get_bound(py).call1((3, 4))?;
            Ok(())
        })
    }
}

Which will result in

---- dag_circuit::test::test_push_back stdout ----
thread 'dag_circuit::test::test_push_back' panicked at crates/circuit/src/imports.rs:46:18:
called `Result::unwrap()` on an `Err` value: PyErr { type: <class '_frozen_importlib._DeadlockError'>, value: _DeadlockError("deadlock detected by _ModuleLock('qiskit.circuit.quantumcircuit') at 4318559312"), traceback: Some(<traceback object at 0x1025e8ac0>) }

---- circuit_data::test::import_clbit stdout ----
thread 'circuit_data::test::import_clbit' panicked at crates/circuit/src/imports.rs:46:18:
called `Result::unwrap()` on an `Err` value: PyErr { type: <class '_frozen_importlib._DeadlockError'>, value: _DeadlockError("deadlock detected by _ModuleLock('qiskit.circuit.quantumcircuit') at 4318559312"), traceback: Some(<traceback object at 0x1027d5a40>) }

---- circuit_data::test::import_circ stdout ----
thread 'circuit_data::test::import_circ' panicked at crates/circuit/src/imports.rs:46:18:
called `Result::unwrap()` on an `Err` value: PyErr { type: <class '_frozen_importlib._DeadlockError'>, value: _DeadlockError("deadlock detected by _ModuleLock('qiskit.circuit.quantumregister') at 4318460096"), traceback: Some(<traceback object at 0x10437f780>) }

That is a python issue that we, unfortunately, won't be able to get around easily.

That being said, using the thread safe OnceLock seems sensible to me for future reference. It doesn't seem to affect utility scale, (I haven't been able to test other things due to some ASV failures, fixed by #13781.) I will abstain from adding this to the merge queue in case @mtreinish's questions haven't been answered.

[jakelishman]

I could well be wrong, but my impression here is that the CPython import system still should be safe under thread contention, or the free-threaded build of it couldn't work even for pure-Python programs.

I feel like the deadlock might be more related to our own code. OnceLock isn't any more re-entrant-safe than GILOnceCell (it's a big warning on both their pages), so I suspect that the trouble is that our imports within _accelerate are fundamentally non-thread-safe (makes sense - they are circular), and when import qiskit is done from single-threaded Python, it's initialising us in a safe order, whereas when we're attempting to run Python-space tests hitting the internals of our Python extension module without having called its PyInit_accelerate in a proper order, we're basically trying to use a library without having called its set-up code. It's probably just decent luck that we get as far as we do.

Running the tests in single-threaded mode might be enough to prevent deadlocks, but I suspect that even that isn't truly the problem; I could imagine a situation say where an accelerate function used the CPython extension functionality to take the module object as an argument, but since we haven't actually initialised the module object, it'd explode. I think we might need some way to ensure that we're cleanly import qiskiting before we run any tests, threaded or not.

[jakelishman]

A quick thought: perhaps we could make use of the Python site to insert an import qiskit as part of the Python initialisation?

This solution can be improved/reformulated

[kevinhartman]

A quick thought: perhaps we could make use of the Python site to insert an import qiskit as part of the Python initialisation?

Trying this locally, it works 🙂.

I created a sitecustomization.py file which imports qiskit, and added it to the PYTHONPATH in tools/run_cargo_test.py. Is this what you'd expect, or is there something more clever we can do? Where should we keep such a file within the repo?

[jakelishman]

In an ideal world, I was hoping we could just inject it in memory, but if not, then sticking the file in a directory in tools, like tools/pyo3-tests and poking that into the site path would be fine.

In that universe, does GILOnceCell work again?

[jakelishman]

Just pointing out here: I think the other trouble that Sasha was having with tests is pretty fatal to the PyO3 testing in the current manner. cargo test compiles separate binaries to the _accelerate.abi3.so shared object library that you get from import qiskit, so Rust structs created natively within the test can't interact with Python - it's the same ABI problem that makes C/Python interop impossible for us in the short term.

Anything that actually needs to interact with Python space bidirectionally will fail in the test suite because of the ABI mismatch, I think.

[kevinhartman]

This is also something that @raynelfss was experiencing, and we came to the same conclusion. I suppose the issue is mostly extracting a pyclass originating from the Python dylib into a Rust struct.