rclone sync --create-empty-src-dirs to b2 doesn't create empty directories

[gentoo-root]

What is the problem you are having with rclone?

I have a crypt remote called b2enc over a b2 remote called b2. rclone sync --create-empty-src-dirs to b2enc seems to have no effect on empty directories. When I rclone mount b2enc:, these directories don't show up. With rclone 1.68.2, sync silently ignores them and doesn't create them on remote. With rclone 1.69.1, sync prints messages like these, but still the directories are not created:

2025/02/22 13:16:35 INFO  : DIRECTORY/NAME: Making directory

What is your rclone version (output from rclone version)

rclone v1.69.1
- os/version: arch (64 bit)
- os/kernel: 6.12.9-arch1-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.0
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Arch Linux, 64-bit

Which cloud storage system are you using? (e.g. Google Drive)

Backblaze

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

exec rclone sync --create-empty-src-dirs --links --filter-from ~/.config/rclone/filter-home.txt --fast-list --progress --transfers 12 --verbose ~ b2enc:home

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2025/02/22 14:20:08 DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "sync" "--create-empty-src-dirs" "--links" "--filter-from" "/home/max/.config/rclone/filter-home.txt" "--fast-list" "--progress" "--transfers" "12" "-vv" "/home/max" "b2enc:home"]
2025/02/22 14:20:08 DEBUG : Creating backend with remote "/home/max"
2025/02/22 14:20:08 DEBUG : Using config file from "/home/max/.config/rclone/rclone.conf"
2025/02/22 14:20:08 DEBUG : Creating backend with remote "b2enc:home"
2025/02/22 14:20:08 DEBUG : Creating backend with remote "b2:backup-XXXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXX"

2025/02/22 14:20:10 DEBUG : .ssh: Excluded
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s
2025/02/22 14:20:10 DEBUG : .cache: Excluded
<...>
2025/02/22 14:26:04 INFO  : EMPTY/DIRECTORY/NUMBER/ONE: Making directory
Transferred:      581.914 MiB / 581.914 MiB, 100%, 2.636 MiB/s, ETA 0s
Checks:            532878 / 542881, 98%
Transferred:            9 / 9, 100%
Elapsed time:      5m54.0s
2025/02/22 14:26:04 INFO  : EMPTY/DIRECTORY/NUMBER/TWO: Making directory
Transferred:      581.914 MiB / 581.914 MiB, 100%, 2.636 MiB/s, ETA 0s
Checks:            532878 / 542882, 98%
Transferred:            9 / 9, 100%
Elapsed time:      5m54.0s
<...>
2025/02/22 14:26:23 DEBUG : Encrypted drive 'b2enc:home': Waiting for transfers to finish
Transferred:      581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4s
2025/02/22 14:26:23 DEBUG : Waiting for deletions to finish
Transferred:      581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4sTransferred:            581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4s
2025/02/22 14:26:23 INFO  : 
Transferred:      581.914 MiB / 581.914 MiB, 100%, 742.333 KiB/s, ETA 0s
Checks:            679489 / 679489, 100%
Transferred:            9 / 9, 100%
Elapsed time:      6m13.4s

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[kapitainsky]

Empty directories are not supported by B2.

You can switch to S3 compatibility layer and use https://rclone.org/s3/#s3-directory-markers

[gentoo-root]

Empty directories are not supported by B2.

I see... Thanks for the information. Can rclone do something on its side, e.g., upload directory markers like on S3 (or files with the .rclonedir extension if the trailing slash is not supported for filenames on B2)?

UX-wise, an error message when trying to use --create-empty-src-dirs on B2 would make sense, and the new behavior in 1.69.1 produces a lot of noise printing "Making directory" for all empty dirs on every sync (and the message is also misleading).

[gentoo-root]

You can switch to S3 compatibility layer

It seems it's not a drop-in solution, because it started reuploading and replacing many terabytes of my data, and I'm not sure why. After a file is uploaded via b2, the b2 remote shows its mtime, but the s3 remote shows the upload time. After the same file is reuploaded via s3, they switch: b2 shows the upload time, and s3 shows the actual mtime.

Edit: that's why:
For s3: The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns.
For b2: The modification time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard.

[kapitainsky]

For s3: The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns.
For b2: The modification time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard.

You could use --modify-window flag to rectify it. Sounds like 1ms should work here. But the best is to test.

https://rclone.org/docs/#modify-window-time

After a file is uploaded via b2, the b2 remote shows its mtime, but the s3 remote shows the upload time. After the same file is reuploaded via s3, they switch: b2 shows the upload time, and s3 shows the actual mtime.

Or maybe not.. Indeed it looks like you have to plan which API you are using from the start.

[gentoo-root]

Sounds like 1ms should work here.

No, not at all, it's not a matter of 1 ms, the upload timestamp and the mtime of the file can be years or even decades apart. When the file is uploaded via the b2 remote, b2 shows mtime, and s3 shows the upload date. When I start syncing via the s3 remote, rclone doesn't see the mtimes set in X-Bz-Info-src_last_modified_millis, because it queries X-Amz-Meta-Mtime, so it starts reuploading all files. Even worse, as I have versions enabled for my bucket, the paycheck will double until the old versions are cleaned up.

Indeed it looks like you have to plan which API you are using from the start.

Yeah, better luck next time 😅

Back on topic, there are two things that can be done on rclone side: directory markers for b2 (I guess, it should be doable, because I see b2 understands directory markers created with s3), and better UX for the messages (may be not needed if directory markers are implemented). What do you think?

[kapitainsky]

directory markers for b2

https://www.backblaze.com/docs/cloud-storage-files#file-names

"File names cannot start with /, end with /, or contain //."

Which means that it can not use the same method as used for S3 where marker ends with /.

Somebody would have to investigate it further I guess.

I wonder if creating some hidden empty file would be acceptable solution.. It would "create" these dirs.

[gentoo-root]

Which means that it can not use the same method as used for S3 where marker ends with /.

Hmm, I wonder then:

1. How S3 API creates those markers on B2? Maybe "File names cannot start with /, end with /, or contain //" is a limitation of B2 API, not the underlying storage, though.
2. How I see empty directories via B2 API after the markers were created with S3 API?

Which means that it can not use the same method

Even if so, there are still two alternatives possible:

1. Instead of a file called dir/, create dir.rclonedir as a marker.
2. Create the .rclonedir file inside the dir (as you suggsted).

(For this example, naming was chosen to resemble *.rclonelink stubs for symlinks; I don't insist on these particular names.)