Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node doesn't retry heartbeats #3190

Closed
roman-khimov opened this issue Mar 3, 2025 · 0 comments · Fixed by #3193
Closed

Node doesn't retry heartbeats #3190

roman-khimov opened this issue Mar 3, 2025 · 0 comments · Fixed by #3193
Assignees
@End-rey
Labels
bug Something isn't working I4 No visible changes neofs-storage Storage node application issues S4 Routine U1 Critically important to resolve quickly
Milestone
v0.45.1

Comments

@roman-khimov
Copy link
Member

Expected Behavior

Node sending heartbeat successfully.

Current Behavior

error	netmap/process_peers.go:133	can't invoke netmap.UpdatePeer	{"error": "failed to submit notary request: The memory pool is full and no more transactions can be sent (-502) - no space left in the memory pool"}

Possible Solution

Retry. Similar to #3128. -502 is a temporary error.

Steps to Reproduce (for bugs)

Add >300 nodes to the cluster.

Your Environment

  • Version used: 0.45.0
@roman-khimov roman-khimov added bug Something isn't working I4 No visible changes neofs-storage Storage node application issues S4 Routine U1 Critically important to resolve quickly labels Mar 3, 2025
@roman-khimov roman-khimov added this to the v0.45.1 milestone Mar 3, 2025
@roman-khimov roman-khimov mentioned this issue Mar 3, 2025
Closed
End-rey added a commit that referenced this issue Mar 3, 2025
@End-rey
ir/netmap: add retries to process update peer notification
627163c 
Retry sending approval tx for process update peer notification.

Closes #3190.

Signed-off-by: Andrey Butusov <andrey@nspcc.io>
@End-rey End-rey mentioned this issue Mar 3, 2025
Merged
End-rey added a commit that referenced this issue Mar 3, 2025
@End-rey
morph: add retries to sending notary request
1976a5d 
Closes #3190.

Signed-off-by: Andrey Butusov <andrey@nspcc.io>
@roman-khimov roman-khimov mentioned this issue Mar 4, 2025
Closed
End-rey added a commit that referenced this issue Mar 4, 2025
@End-rey
morph: retry only on temporary errors
daacd6b 
In #3190 added retries for sending notary requests that was received from Notary
service. Now retry only if there is `neorpc.ErrMempoolCapReached` error.

Signed-off-by: Andrey Butusov <andrey@nspcc.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@End-rey End-rey

Labels
bug Something isn't working I4 No visible changes neofs-storage Storage node application issues S4 Routine U1 Critically important to resolve quickly
Projects
None yet
Milestone
v0.45.1
Development

Successfully merging a pull request may close this issue.

Add retries to process update peer notification nspcc-dev/neofs-node
2 participants
@roman-khimov @End-rey