Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ignore broken objects when migrating to metabucket #3203

Closed
cthulhu-rider opened this issue Mar 5, 2025 · 1 comment
Closed

Ignore broken objects when migrating to metabucket #3203

cthulhu-rider opened this issue Mar 5, 2025 · 1 comment
Assignees
@cthulhu-rider
Labels
enhancement Improving existing functionality I4 No visible changes neofs-storage Storage node application issues S4 Routine U1 Critically important to resolve quickly
Milestone
v0.45.1

Comments

@cthulhu-rider
Copy link
Contributor

Is your feature request related to a problem? Please describe.

neofs-node/pkg/local_object_storage/metabase/version.go

Line 114 in 951938a

func migrateFrom3Version(_ *DB, tx *bbolt.Tx) error {
returns an error on any object which is not a valid protobuf. This causes the SN to shutdown due to storage init failure. In this approach, one corrupted object prevents all other valid ones from being served

Describe the solution you'd like

if the next object is not decoded, log its address (info) and BLOB (debug), skip and move on to the next one. Then the sick object will be cured by administration in one way or another, while healthy ones will be more quickly accessible to SearchV2 protocol

Describe alternatives you've considered

consider the metabase broken in this case and require resync. Generally legit, but in some cases may be too aggressive and time-spending

Additional context
@cthulhu-rider cthulhu-rider added enhancement Improving existing functionality neofs-storage Storage node application issues labels Mar 5, 2025
@cthulhu-rider cthulhu-rider added this to the v0.46.0 milestone Mar 5, 2025
@cthulhu-rider cthulhu-rider self-assigned this Mar 5, 2025
@roman-khimov roman-khimov modified the milestones: v0.46.0, v0.45.1 Mar 5, 2025
@roman-khimov roman-khimov added U1 Critically important to resolve quickly S4 Routine I4 No visible changes labels Mar 5, 2025
@cthulhu-rider
Copy link
Contributor Author

do not migrate objects from removed containers

cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Ignore invalid protobufs on metabucket migration
85616ae 
Previously, migration func storing objects' metadata in the metabucket
failed on any invalid object protobuf. With this, any invalid object
prevented all valid ones to be served over SearchV2 protocol.

This makes migrator to write info about broken object to the log, ignore
it and go next.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Unify header metadata processing
9fa1c6a 
Unify the object header check before writing to the meta bucket and the
actual write for `Put` and migration routine. Also split header errors
from the BoltDB transaction ones.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Assert object header does not exceed the size limit
335aa84 
Object headers must not exceed 16K. This should be checked before saving
to the metabucket. This also ensures that the BoltDB limits are met: any
key should not exceed 32K.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Ignore NeoFS protocol violations on metabucket migration
7dbda76 
Same as 85616ae but for NeoFS protocol
disorders.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Ignore missing containers on metabucket migration
fe63d25 
Same as 85616ae but for missing
containers. Once deleted, the container will never appear again. All its
data should be cleared, including not creating new ones.

Closes #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Ignore invalid protobufs on metabucket migration
e92cbd7 
Previously, migration func storing objects' metadata in the metabucket
failed on any invalid object protobuf. With this, any invalid object
prevented all valid ones to be served over SearchV2 protocol.

This makes migrator to write info about broken object to the log, ignore
it and go next.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Unify header metadata processing
6607b1e 
Unify the object header check before writing to the meta bucket and the
actual write for `Put` and migration routine. Also split header errors
from the BoltDB transaction ones.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Assert object header does not exceed the size limit
205c298 
Object headers must not exceed 16K. This should be checked before saving
to the metabucket. This also ensures that the BoltDB limits are met: any
key should not exceed 32K.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Ignore NeoFS protocol violations on metabucket migration
ac16053 
Same as e92cbd7 but for NeoFS protocol
disorders.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 5, 2025
@cthulhu-rider
node/metabase: Ignore missing containers on metabucket migration
5f70df4 
Same as e92cbd7 but for missing
containers. Once deleted, the container will never appear again. All its
data should be cleared, including not creating new ones.

Closes #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 6, 2025
@cthulhu-rider
node/metabase: Ignore invalid protobufs on metabucket migration
70d625e 
Previously, migration func storing objects' metadata in the metabucket
failed on any invalid object protobuf. With this, any invalid object
prevented all valid ones to be served over SearchV2 protocol.

This makes migrator to write info about broken object to the log, ignore
it and go next.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 6, 2025
@cthulhu-rider
node/metabase: Unify header metadata processing
316c1c9 
Unify the object header check before writing to the meta bucket and the
actual write for `Put` and migration routine. Also split header errors
from the BoltDB transaction ones.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 6, 2025
@cthulhu-rider
node/metabase: Assert object header does not exceed the size limit
bad86f0 
Object headers must not exceed 16K. This should be checked before saving
to the metabucket. This also ensures that the BoltDB limits are met: any
key should not exceed 32K.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 6, 2025
@cthulhu-rider
node/metabase: Ignore NeoFS protocol violations on metabucket migration
dcfcba8 
Same as 70d625e but for NeoFS protocol
disorders.

Refs #3203.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cthulhu-rider cthulhu-rider

Labels
enhancement Improving existing functionality I4 No visible changes neofs-storage Storage node application issues S4 Routine U1 Critically important to resolve quickly
Projects
None yet
Milestone
v0.45.1
Development

No branches or pull requests
2 participants
@roman-khimov @cthulhu-rider