orphaned objects generated when compacting repository

[88mm]

Once a week my Borg backup script adds a "borg compact" to the daily Borg backup execution. The "borg compact" gives a "rc 0" exit status. The next command is a "borg check" which finds an "Index object count mismatch." I subsequently run a "borg check --show-rc --repair" by hand to fix the issue and receive an "orphaned objects found" message. After that the daily "borg check" (without --repair) exits without finding any errors for a week until the next "borg compact". This has happened for the last three weeks in a row. I have been using this Borg repository for about a year so the daily "borg prune" always deletes an archive. Any suggestions for fixing this issue would be appreciated.

With all the arguments removed here what my daily Borg backup script does:

borg info
borg create
borg prune
borg compact  (but just once a week)
borg check

Here is part of the backup output showing the appearance of the object count mismatch:

------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:             -413.92 GB           -298.43 GB            -30.74 MB
All archives:                6.56 TB              4.73 TB            276.72 GB

                       Unique chunks         Total chunks
Chunk index:                 5059073             93169648
------------------------------------------------------------------------------
terminating with success status, rc 0

2022-09-04T00:06:55 Compacting repository

compaction freed about 222.65 MB repository space.
terminating with success status, rc 0

2022-09-04T00:07:01 Checking repository

Index object count mismatch.
committed index: 5059074 objects
rebuilt index:   5059655 objects
ID: 37035278f35bc93f9ee30f1a9e9993937b78e3901e7b1ade78d208182cef0748 rebuilt index: (980, 54564816)  committed index: <not found>     
ID: 4e7c9ae2a8c03f6d261ae9526a9b499561d4115c4b018e4576b83ce68495d886 rebuilt index: (980, 76805353)  committed index: <not found>   
    ...

Here is the manual fix:

# borg check --show-rc --repair
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: YES
581 orphaned objects found!
Archive consistency check complete, problems found.
terminating with success status, rc 0
# borg check --show-rc 
terminating with success status, rc 0

I am using Python-3.10.6 and borgbackup-1.2.1.
This Borg repository was originally created under borgbackup-1.1.17. There were no unresolved issues when migrating to borgbackup-1.2.1. I did follow the instructions in the Borg changelog notes. I added the "borg compact" after upgrading to borgbackup-1.2.1.

If I should provide more information or my entire backup script let me know.

(As a last resort I am considering stopping the use of this Borg repository and starting a new one. I would keep this old repository for a half year if I need an old restore.)

[ThomasWaldmann]

We already have a ticket for this, please locate it and add new information there.

[infectormp]

#6781

[ThomasWaldmann]

Closing this as a duplicate.

[ThomasWaldmann]

@88mm BTW, if you could provide a "reproducer script" that starts from nothing uses as little backup data as possible and is able to reproduce the issue (anywhere), that would be super useful to find the root cause and fix it.

[88mm]

I unsuccessfully tried to reproduce this bug in a new/small repo. I temporarily reverted to borgbackup-1.1.17 and then "upgraded" the repo to 1.2.1. I implemented the changes I made to the "borg create" arguments: I switched from --compression=auto,zstd,10 to --compression=auto,zstd,12 (and switched to --noflags from --nobsdflags).

[ThomasWaldmann]

@88mm ok, thanks for trying to reproduce with a small repo.