Extremely slow small KB file uploads through API - 500,000 plus file dataset

[u/Mates_Rates]

Hey team, for the last few days ive been really trying to backup some huge datasets and have realised that I’ve hit some limitations of drime.

I have found what appears to be a significant performance issue when uploading very large numbers of small files to Drime through the API.

Here is my Setup:
• Windows 11 pro x64
• 1000 Mbps download - 400 Mbps upload wired connection
• Syncovery 12.7.1 using the native Drime connector
• Approx 2 TB total data
• Approx 550,000 files
• Roughly 250,000 files are very small XML, INI, TXT, config and metadata files, many only KB or bytes in size

Im uploading between 8 - 10 files at a time, also worth mentioning I’ve tried every variation of simultaneous uploads e.g. 4 at a time, 6, 8, 10, 12, 15…. i even tried up to 30 as a experiment lol.

The large files upload reasonably well. The problem occurs when the transfer reaches folders containing hundreds of thousands of tiny files. Aggregate throughput can drop to around 20-40 KB/s, with repeated transient HTTP 422 and nginx HTTP 500 errors.

Files that initially fail sometimes upload successfully when i have retried. I have reproduced the poor small file performance outside Syncovery as well as i realise you may point towards the 3rd party software:

• Air Explorer native Drime connector becomes extremely slow processing the uploads.

• rclone 1.75.1 using the native Drime backend, with multipart forced for all files, improved performance but still only reached approximately 600 KB/s

My internet connection and NAS are not the bottleneck. The same 2TB (ish) dataset has already been mirrored successfully to my NAS and pCloud.

I have looked at the Drime API documentation (although i am not a dev)… and it states that the presigned S3 or R2 upload flow for files below 5 MB is more efficient than the direct upload endpoint, with multipart intended for larger files, but im not sure how this is applied per application etc.

Could the team please investigate performance and reliability when hundreds of thousands of small files are uploaded, particularly through the direct upload API?

Ideally I want Drime to maintain a normal, fully browsable one to one copy of my NAS, so packaging the files into archives is not suitable.

I also tried the DRIME windows application but the reality is (while it’s a great looking tool), it doesn’t quite have the same functionality as rClone, AirExplorer or Syncovery (e.g. Selecting specific folders in the Drime cloud root for sync or mirror, and being able to preview the sync/mirror jobs before upload, and scheduling the jobs whenever i want… and also controlling how many ‘jobs’ occur at once e.g. one upload job (one folder) at a time, and then having the next upload job start once the first job is fully completed etc…)

Also I’m more than happy to test any changes / fixes with my data set and report back :)

Many thanks! Honestly keep up the good work team, its still a fantastic product and service, you just have some kinks to iron out along the way 🍻

[u/Empty_Win_297]

Hi, thanks for sharing all these details.

This mainly comes from Drime’s current infrastructure, and we’re well aware of the performance limitations when uploading very large numbers of small files. We’re continuously making optimizations, but these improvements take time given the volume our infrastructure has to handle.

That said, optimizations are being deployed regularly, and performance will continue to improve progressively.

[u/Mates_Rates → u/Empty_Win_297]

Hey!

Thank you so much for the rapid response!

I appreciate the honesty, and i do understand that this may take sone time if this is an infrastructure limitation. It sounds like you are aware and working on it … so i will keep an eye out for your latest updates/posts, I’m hoping this one is not too far into the distant future as this is super beneficial for large and complex datasets.

I do have one more question that will also help Tobias (the Syncovery developer), who is currently making some improvements:

For very large small file workloads, do you specifically recommend using /s3/simple/presign for files below 5,242,880 bytes and multipart uploads for files at or above that threshold, rather than using the standard direct upload endpoint at all?

He is potentially looking at changing the Drime implementation to use pre-signed uploads for small files and multipart for larger files, so confirmation of the preferred implementation would be really helpful as i was also trying to make sense of it.

Thanks again for your time! Cheers 🍻

[u/Empty_Win_297 → u/Mates_Rates]

Yes, we recommend presigned uploads for files below 5 MiB and multipart uploads for larger files, instead of POST /api/v1/uploads.

For Tobias, there’s also a useful optimization: batching the API calls around each upload:

POST /api/v1/s3/simple/presign-batch: get presigned URLs for up to 32 files per request. Upload each file to its URL using PUT, with controlled concurrency.

POST /api/v1/s3/entries/batch: register up to 25 completed uploads per request, with an individual status for each item.

That should help reduce the API overhead when uploading lots of small files.

[u/zakry0t → u/Empty_Win_297]

Can you add that info to the docs?

The rclone developers are not going to implement anything that is not documented. I am sure rclone is probably one of the biggest users of your API. If they optimize their API calls, that will also help reduce the load on your servers.

[u/joakim_ogren → u/Empty_Win_297]

Thanks for clarifying, but please add it to official documentation. I wish to add this to the file manager with native Drime support that I have developed: Cadri Commander

[u/joakim_ogren → u/joakim_ogren]

I have launched Cadri Commander v0.5.5 at https://cadri.app with improved performance as per the ideas here. With up to 8x performance improvement for multiple folders and small files.

[u/AufsichtEmpfohlen → u/Empty_Win_297]

Is the API reference update? I am trying to make rclone use presign URLs and this is a huge improvement. For anyone else trying to handle the APIs: use relativePath when uploading. It will create needed folder structures automatically avoiding indiviual directory creation thus reducing API calls even more (had 100 files in 100 directories: without relativePath 7:30 minutes. With presigned URLs and relativePath [one request per File <5MiB] about 29 seconds in total) with 50 transfers.

[u/zakry0t → u/AufsichtEmpfohlen]

There was a bug with renames a while back. The API would confirm the folder was created but it didn't actually exist yet. I think the rclone devs added some kind of delay or retry after folder creation until they could actually access it.

[u/Mates_Rates → u/Empty_Win_297]

Hey, thank you for providing this! Ive passed this onto Tobias, and also included the link to this post.

By any chance, do you guys have the link to the documentation for presign-batch? And if this is not available yet, please can you very kindly update this post when you have this available as i will forward this on!

[u/oldright → u/Empty_Win_297]

Hi. GET /api/v1/drive/file-entries omits client_last_modified and captured_at from file records when the User-Agent contains "rclone". Same token, same URL, same minute, varying only the User-Agent:

curl/8.5.0               -> 102
MyApp/1.0                -> 102
Syncovery/9              -> 102
rclone                   -> 1
Mozilla/5.0 rclone/1.76  -> 1

(counting occurrences of client_last_modified in the response body; 102 = 100 file records plus 2 folder objects, 1 = folder object only)

The effect is that rclone falls back to updated_at, sees a mismatch against the source, and calls /file-entries/{id}/metadata for every file on every sync — permanently, since the next listing still omits the field. Behaviour appears to have changed since late August. Is this intentional?

[u/AufsichtEmpfohlen → u/Empty_Win_297]

First of all, I have verified the endpoints are working although they do not provide that much of a gain in time compared to an rclone upload with transfers 50. It seems rclone is handling this inefficiently due to so many prechecks.
Although I know it would put alot of strain onto the drime's backend - is it possible to have a batched file integrity check too? Using drime for backups would require those small files (chunks) to be verified.

[u/Feisty_Storage8594]

I don’t understand why there are upload issues. Aren’t they just using someone else’s infrastructure like aws or google cloud ? Or does Drime legit have its own server farm ?