[Dev Tools] Failing hardhat tests after updating to the latest zksync plugin versions

[mmv08]

Team or Project

Safe

Environment

Testnet

Select the Dev Tool you are using

Hardhat Plugins

Provide the version of the tool (if applicable)

No response

Provide a brief description of the functionality you're trying to implement and the issue you are running into.

We had a flaky zksync test that would fail 40% of the runs (more likely when running on a slower CPU). We tried to fix it by updating zksync plugins to the latest versions, but our ZkSync tests are failing now.

There are multiple types of failures:

• locally: the tests start running, but after 20-30 successful tests, they fail. Screenshot attached.
  
  Sometimes there's this error:

  1) DebugTransactionGuard
       fallback
         should revert on fallback with value:
     ProviderError: InMemoryNode lock is poisoned: poisoned lock: another task failed inside
      at HttpProvider.request (/Users/mmv/Projects/safe/safe-contracts/node_modules/hardhat/src/internal/core/providers/http.ts:107:21)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async DeploymentsManager.revertSnapshot (/Users/mmv/Projects/safe/safe-contracts/node_modules/hardhat-deploy/src/DeploymentsManager.ts:1452:30)
      at async /Users/mmv/Projects/safe/safe-contracts/node_modules/hardhat-deploy/src/DeploymentsManager.ts:336:29
      at async Context.<anonymous> (/Users/mmv/Projects/safe/safe-contracts/test/guards/DebugTransactionGuard.spec.ts:45:17)

• CI: the node fails to start. sample run

Pull request with the change: safe-global/safe-smart-account#898

To reproduce:

1. Clone the repo
2. npm i && npm run test:zk

Repo Link (Optional)

safe-global/safe-smart-account#898

Additional Details

No response

[popzxc]

Hello! Thanks for the report. I think there might be a combination of problems here.

1. Node not starting in CI. I think it may be related to the problem with OpenSSL. See the issue -- it should be resolved in the next release this week.
2. A poisoned lock error may happen when in-memory node (now anvil-zksync) executor thread panics and it gets detected by the API thread. It shouldn't happen, it's certainly a bug in the node. noNetwork error may be related to the node crash as well. We will investigate, the issue is already picked up.

If you can pinpoint which tests causes the node to crash locally, it can help with the investigation.

[mmv08]

If you can pinpoint which tests causes the node to crash locally, it can help with the investigation.

Unfortunately, I think the issue only occurs when all the tests are run together. If I run any failing test in isolation, it passes.

[popzxc]

Huh, that's interesting. It might mean that there is some race in the API when transactions are being sent concurrently. We will look into this.

[itegulov]

Hello! There was indeed a race in anvil-zksync, thanks for bringing this to our attention. anvil-zksync v0.2.4 has been released, it should have both bugs fixed. Please try re-running the workflow. Locally you can delete the release cache by running:

• Linux: rm ~/.cache/hardhat-nodejs/zksync-memory-node/latestRelease.json
• macOS: rm ~/Library/Application Support/hardhat-nodejs/zksync-memory-node/latestRelease.json

After which v0.2.4 will be downloaded on your next hardhat test run

[mmv08]

Thank you! it is working now

[mmv08]

just fyi the node cache was in ~/Library/Caches/hardhat-nodejs/zksync-memory-node/latestRelease.json for me