[ZK Stack] - Reverting executed batch

[Teamvalid]

Team or Project

Validium

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.82.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

Our chain is using holesky network as a L1, in the recent update holesky had some issues due to fix transaction on our chain were not been not sent to L1. Transactions were stuck on "sending" status for couple of days. So we decided to rollback to last executed stage.

Upon using block reverters print-suggested-values, recommneded values are

Suggested values for reversion: SuggestedRevertValues {
    last_executed_l1_batch_number: L1BatchNumber(
        26678,
    ),
    nonce: 61636,
    priority_fee: 1000000000,
}

But last executed batch is 26685

We used these 2 commands clear-failed-transactions and rollback-db to fix and 26685 for --l1-batch-number flag. Chain state rolled back but it didnt fixed the problem. Plus, we now getting these errors

ERROR zksync_node_framework::service: Task eth_tx_manager panicked: called `Result::unwrap()` on an `Err` value: EthereumGateway(EnrichedClientError { inner_error: ParseError(Error("invalid type: string \"couldn't complete the request, try again later\", expected struct ErrorObject", line: 1, column: 80)), method: "get_block_number", args: {} })


WARN EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: eth_sender error EthereumGateway(EnrichedClientError { inner_error: Call(ErrorObject { code: ServerError(-32000), message: "missing trie node c18ce71676af753fbef66a3991d81a3d4451f785868bb1c2b8461a8e45c7652d (path ) state 0xc18ce71676af753fbef66a3991d81a3d4451f785868bb1c2b8461a8e45c7652d is not available", data: None }), method: "get_transaction_count", args: {"account": "0xda6e111c2b23df9dcadb569f9fed9860a8e322e0", "block": "Number(3418062)"} })


Caused by:
0: JSON-RPC request get_block_number() failed: Parse error: invalid type: string "couldn't complete the request, try again later", expected struct ErrorObject at line 1 column 80
1: Parse error: invalid type: string "couldn't complete the request, try again later", expected struct ErrorObject at line 1 column 80
2: invalid type: string "couldn't complete the request, try again later", expected struct ErrorObject at line 1 column 80)), ("eth_tx_manager_layer", ResourceLacking { id: ResourceId(TypeId(0x98bb2403017f09894553b1c947696a48)), name: "common/gas_adjuster" }), ("l1_gas_layer", ResourceLacking { id: ResourceId(TypeId(0x98bb2403017f09894553b1c947696a48)), name: "common/gas_adjuster" }), ("mempool_io_layer", ResourceLacking { id: ResourceId(TypeId(0x222b30a45fd396752dcd6f95d60a9197)), name: "common/sequencer_fee_input" }), ("state_keeper_layer", ResourceLacking { id: ResourceId(TypeId(0xe1ed1730f8d2bb2df1761533d2a1b43c)), name: "state_keeper/io" }), ("tx_sender_layer", ResourceLacking { id: ResourceId(TypeId(0xcdbd35e0af7d92ebc53b73c66d3bb06d)), name: "common/api_fee_input" }), ("web3_http_server_layer", ResourceLacking { id: ResourceId(TypeId(0x5918940a777673edfece03a78677a174)), name: "api/tx_sender" }), ("web3_ws_server_layer", ResourceLacking { id: ResourceId(TypeId(0x5918940a777673edfece03a78677a174)), name: "api/tx_sender" })]

We are using infura as RPC provider. Now we are thinking to rollback to 26678 batch, which is a executed batch but block reverted docs suggested that it is ultra dangerous (Flag that allows to roll back already executed blocks. It's ultra dangerous and required only for fixing external nodes.). This is our blob operator address - https://holesky.etherscan.io/address/0xb2c2c467e0ef64070b56eec0ae499174912de565
It has multiple failed transactions.

So we are unsure on how to fix this.

Expected Behavior

Transactions should settle on L1

Current Behavior

Servers crashing in every 5 minutes, transactions are stuck on "sending" status.

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[Teamvalid]

Closing this as some of the error has been fixed.