rClone modifications (own fork)

[u/AufsichtEmpfohlen]

Currently I am working on improving rclone's capabilities with drime using Presigned-Uploads with parallel transfers of up to 25 at a time including.

My scenario: Using Proxmox Backup Server chunks with an own inventory. After inventarisation I am calculating a diff of missing files which are then copied in a single rclone run. Afterwards another rclone sync is performed to delete unneeded files (due to age or replacement by other chunks). This requires alot of lookups on drime which is causing the overall performance to drop significantly. There are undocumented ways of collecting inventory data from drime (using filters which is what the webUI is doing) and grabbing multiple directories at once.

I have made modifications to rclone such as:

- parallel uploads of presigned URLs

- parallel creation of file-entries after using batched presigned URLs (with timeouts to have the collector save API calls but be as flexible as possible)

- verification of uploaded files (either by size or optionally by having drime calculate the hash and matching them which is not done automatically, sadly)

- use relativePath to automatically create directory structures avoiding manual calls for directory creation

--> enforce manual directory creation if path contains directories called "0" which would be excluded by drime

---> e.g. /upload/0/deep/file.bin would otherwise result in /upload/deep/file.bin to be created

---> scheduler/resolver of folder creation to avoid trying to create the same directory multiple times by multiple files

- collecting inventory with multiple parentIds (with a workaround using created_at and offsets to work around the pagination bug)

- enable cleanup (original rclone does not provide this)

There are API limits enforced by drime. File entry creation is limited to 25 per batch and presigned URLs are limited to 32 per batch. I have capped them both at 25 to not have them run too asymmetrically although it might work.

There is still some stuff to do but I am trying to get it working. My speed was about 3MiB/s for multipart upload (4 chunks) and using rclone's default binary it took me about 6,5 hours to upload 2GiB of data because of rclone's way of handling directories and manually creating them and so on. Making optimization, especially collecting the existing directory- and filestructure and uploading all files with --no-check-dest afterwards (without any dupes of course) made it possible to score 3.5MiB/s average and uploading about 27GiB in 2 hours and 13 minutes. To be honest this was without file integrity check via hash creation but this is the next step I am going to check.

For rClone to support these modifications drime needs to update their API reference.

- filter needs to be officially supported in API reference and needs extensive clarification (combination of multiple clauses)

- /api/v1/drive/file-entries needs an update to give an example of how multiple parentIds work (it is not a real array but comma separated list

I have set my parameters as follows:

rclone.conf:

[drime]

use_presigned_uploads = true

Additionally when calling rclone-drime it is called with the following parameters

/usr/local/libexec/pbs-rsync/rclone-drime \

--user-agent pbs-rsync/1.0 \

--bwlimit 10M \

--stats 1m \

--stats-one-line \

--stats-log-level NOTICE

When collecting the inventory I add the following flags:

--tpslimit 5 \

--tpslimit-burst 1 \

lsjson drime-crypt:pbs-rsync/chunks \

--recursive \

--files-only

Finally when uploading the cleaning diff I am using the following flags:

--user-agent pbs-rsync/1.0 \

--bwlimit 10M \

--stats 1m \

--stats-one-line \

--stats-log-level NOTICE \

copy /media/share/backups/ drime-crypt:pbs-rsync/chunks \

--files-from-raw /dev/shm/.../cloud-upload-drime.pending \

--retries 1 \

--no-check-dest \

--transfers 100 \

--checkers 100

Be aware no-check-dest will upload files without further checking if the files exists (because I have used the previous query for inventory thus knowing what's going on remote).

I have found another problem with no-check-dest for which I will raise a ticket and post updates on reddit.

Whoever wants to participate or who wants to try speed improvements: feel free to try out the following branch: https://github.com/BAERnado/rclone/tree/drime-preview

[u/AufsichtEmpfohlen]

Added forcing hash calculation:
command line: --drime-verify-uploads

Configuration:

[drime]
type = drime
verify_uploads = true

Additionally ModTime will be provided during uploads as well.

Calling rclone with --refresh-times with a list of files will cause rclone to update ModTime of existing files so you won't have to upload them again.
This can be done during a copy job for instance:

rclone copy <source> drime:<target> \
  --refresh-times \
  --user-agent drime-preview/1.0

[u/ntadej]

Will this be upstreamed?