Optimistically queueing unsafe L2 execution payload

[yuanwj210]

version:op-node-v1.7.3
system:ubuntu20.04

After the update to version 1.7.3, the op-geth is stuck, the op-node shows the following log, and the wait for L1 is very long

t=2024-04-16T01:54:07+0000 lvl=warn msg="failed to serve p2p sync request" serve=payloads_by_number peer=16Uiu2HAku-1938 remote=/ip4/173.231.40.186/tcp/9222 req=118801905 err="peer requested unknown block by number: not found"
t=2024-04-16T01:54:07+0000 lvl=info msg="Received signed execution payload from p2p" id=0x93096ac52bc9b3aeacd2b87e7cdeb10b3c5011c6a13207f23eedb391648dd688:118816835 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:07+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x93096ac52bc9b3aeacd2b87e7cdeb10b3c5011c6a13207f23eedb391648dd688:118816835
t=2024-04-16T01:54:09+0000 lvl=info msg="Received signed execution payload from p2p" id=0x67501cf757210241e214083d2eb32bb3278ba080a657816dd4057764e62c639f:118816836 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:09+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x67501cf757210241e214083d2eb32bb3278ba080a657816dd4057764e62c639f:118816836
t=2024-04-16T01:54:11+0000 lvl=warn msg="failed to serve p2p sync request" serve=payloads_by_number peer=16Uiu2HAku-1938 remote=/ip4/173.231.40.186/tcp/9222 req=118801861 err="peer requested unknown block by number: not found"
t=2024-04-16T01:54:11+0000 lvl=info msg="Received signed execution payload from p2p" id=0xacb719ca8dd6e89fb6873d9d1939114fdcb5f3af20942b140c4fed0ff1d37d95:118816837 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:11+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xacb719ca8dd6e89fb6873d9d1939114fdcb5f3af20942b140c4fed0ff1d37d95:118816837
t=2024-04-16T01:54:13+0000 lvl=info msg="Received signed execution payload from p2p" id=0xaf2748a592d22fc5d4cae777e634c6b110cd6f3a925b39633b0ab05516e75e1e:118816838 peer=16Uiu2HAmBw2v1bcguvPJakCkqtNMLMMQQ25ZopiThPNB6QjfrZj1
t=2024-04-16T01:54:13+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xaf2748a592d22fc5d4cae777e634c6b110cd6f3a925b39633b0ab05516e75e1e:118816838

[yuanwj210]

@shazow @mslipper @fgimenez

[jwelch-qn]

I am also seeing this at times. Block height will struggle to stay at tip for a while and the logs are full of

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xdc78579b115e5a967ae549af90ab97b1c22a2d8079a85af3e3640fa0205868b9:118944857

and

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=error msg="Payload execution failed" block_hash=0x688b7bebbdacc9a26d5fd2b55152ea49f627a730d27980ae20904a359200685f err="context deadline exceeded"

Apr 19 01:01:36  op-node[37162]: t=2024-04-19T01:01:36+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: context deadline exceeded"

Running version Geth/v1.101311.0-stable-e9a306ba

[wangjiangw]

we have also found this issue, which makes the block syncing latency become bigger ?

[jwelch-qn]

Yes, nodes start to struggle to stay at blockchain height, they generally resolve on their own, but this appears to be a recent behavior.

[wangjiangw]

jwelch-qn

yes, it will recover at some time in the future, and this happens just in recent monthes

[SyedMujebeurRehman]

HI, can any one help, since 3 weeks it is not downloading the blocks

node_1 | t=2025-02-17T10:50:10+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xd253903f413afb1376a22606789eba3ea0e971012a7a543c13ca135b4d97a5eb:22010561

node_1 | t=2025-02-17T10:52:04+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkwquWFxQWbf5KtLfZwbqMDXgLy9dhsbGfNxkgvYvnqzBM
node_1 | t=2025-02-17T10:52:04+0000 lvl=info msg="connected to peer" peer=16Uiu2HAkwquWFxQWbf5KtLfZwbqMDXgLy9dhsbGfNxkgvYvnqzBM addr=/ip4/34.65.154.74/tcp/9003
node_1 | t=2025-02-17T10:52:04+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAkwquWFxQWbf5KtLfZwbqMDXgLy9dhsbGfNxkgvYvnqzBM
node_1 | t=2025-02-17T10:52:04+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm7QJk2sg1fn9G4JLX1m3vR9FmLihuMLoVLF2GhpkkyxZh
node_1 | t=2025-02-17T10:52:04+0000 lvl=info msg="Received signed execution payload from p2p" id=0x5f2a6f5e0b6684fc9eadec1d23faaa413a794ddc167afee7d1121104921c0fd8:22010618 peer=16Uiu2HAkwquWFxQWbf5KtLfZwbqMDXgLy9dhsbGfNxkgvYvnqzBM
node_1 | t=2025-02-17T10:52:04+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x5f2a6f5e0b6684fc9eadec1d23faaa413a794ddc167afee7d1121104921c0fd8:22010618
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkv2SfY9j8G3riJFW4tJbhtfSLhLoqWJjjjhsafQQwtV46
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkvMZeksuCQiqjvXREoLfDBRPUKCEFCcNxFpbiQUwbyqh2
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="connected to peer" peer=16Uiu2HAkv2SfY9j8G3riJFW4tJbhtfSLhLoqWJjjjhsafQQwtV46 addr=/ip4/68.234.2.106/tcp/9222
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAkv2SfY9j8G3riJFW4tJbhtfSLhLoqWJjjjhsafQQwtV46
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmB7JbAJ2GsF27CDERfCJPxEGjyaXgYFrKKMvkoM8CoJBz
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmB7JbAJ2GsF27CDERfCJPxEGjyaXgYFrKKMvkoM8CoJBz addr=/ip4/148.251.52.163/tcp/9222
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmB7JbAJ2GsF27CDERfCJPxEGjyaXgYFrKKMvkoM8CoJBz
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmKFwuY7KDNCSd9SsJ81jga7WQZLmJ1QrzsbGC9nN45aMu
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkyjbz5wngCxgNdySQsPk75CHV9e7MTEN1NLufRyVsy8c5
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="connected to peer" peer=16Uiu2HAkyjbz5wngCxgNdySQsPk75CHV9e7MTEN1NLufRyVsy8c5 addr=/ip4/130.61.117.169/tcp/9222
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAkyjbz5wngCxgNdySQsPk75CHV9e7MTEN1NLufRyVsy8c5
node_1 | t=2025-02-17T10:52:05+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkwbW3wXhnJqpPAo91kL8G1KtJRqNoWH4uCdvK8q6XXgRp
node_1 | t=2025-02-17T10:52:06+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmKFwuY7KDNCSd9SsJ81jga7WQZLmJ1QrzsbGC9nN45aMu addr=/ip4/173.231.43.98/tcp/13509
node_1 | t=2025-02-17T10:52:06+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmKFwuY7KDNCSd9SsJ81jga7WQZLmJ1QrzsbGC9nN45aMu