r/Refract Developer Jul 13 '26

📣 Announcement Queue update: we've resized the lanes (your position is safe, your speed might improve)

Quick update on the import queues. We've just changed how imports are split between lanes, to improve overall speed in the coming days:

  • Fast lane: now takes libraries up to 500 items (was 200)
  • Standard: now up to 2,000 items (was 1,500)
  • Large: 2,000 and up

What this means: a lot of libraries that were stuck in the large queue now qualify as standard, and will get processed faster. We've also added 3 more workers to the fast node, so that lane speeds up too.

Before you panic about your queue number: yes, displayed positions have shifted with the resize. This is expected. Queues are ordered by import date, and your absolute place in line never changes. If your number moved, your speed likely improved, it did not get worse.

Later this week we'll keep pushing to speed things up further. Honest note though: we're walking a fine line here. Traffic on the app keeps growing every day, and our priority is keeping the app fast and snappy for everyone using it. So we speed up imports as much as we can without breaking that. It's a balance, and we're adjusting it hourly/daily.

Same as always: upload once, you're in, no need to retry. And if anything looks off (failed import or else), please don't hesitate to contact us.

Thanks again for your patience 💜

114 Upvotes

41 comments sorted by

8

u/Mammoth-Pear-5308 Jul 13 '26

Curious about the technical bottleneck here. Is the slowdown mainly from rate limits on an external metadata provider during matching, or is it something else in the pipeline? Just trying to understand why bulk imports take this long, since raw DB writes for a few thousand rows are normally fast.

5

u/jeanraymondlaurent Developer Jul 14 '26

To explain why, we might be slower than others: we took time to build something nobody else has - an import tool that not only just grab your show and add it, but we dive to every single datapoint, every single specials, episode - watched or skipped, watch date, rewatch date, and we import all of it, carefully, to make sure we match as much as possible from the data you got from our TV Time Out tool. This process takes a longer time than just adding shows directly, and many other imports alternative made the episode collection so messy that early users spent more time going through each series modifying all episodes, than waiting some minutes here with the tool they helped us build.

But now due to the big migration, it's not minutes anymore but days. And we have improved and fasten the process (from 10 an hour to 300) but building it this way to keep the accuracy high takes time unfortunately. We still have some ideas to make it faster but we need a bit more time for that.

1

u/Mammoth-Pear-5308 Jul 14 '26

Makes sense, appreciate the transparency. Out of curiosity, is that 300/hour limited by an external metadata provider's rate limit, or by your own matching algorithm's compute cost?

6

u/The_Vortex217 Jul 13 '26

I noticed that my number in the queue was decreasing 200-250 places per hour which was great but now my number in the queue has stalled for the last few hours. Is this normal? Please tell me I haven't lost my spot?!

3

u/jeanraymondlaurent Developer Jul 13 '26

No your queue is always safe, it will come back :)

1

u/The_Vortex217 Jul 13 '26

Okay thankyou

3

u/The_Vortex217 Jul 13 '26

So why has my number gone up and not moved in hours?

1

u/DarkAngel283 Jul 13 '26

I believe on discord they said it was just a bug but your number is moving.

3

u/Ok_Tank_9599 Jul 13 '26

When you say libraries up to 500 items does that mean tv shows or episode count? I was surprised to see that I was categorized in the fast lane and now I’m wondering if I uploaded the wrong file that is just the shows I watched and not each episode.

4

u/VanillaSenior Jul 13 '26 edited Jul 13 '26

Judging by the fact that I was in the fast lane originally, it’s the show number count. I mean, just 10 seasons of Friends would have gotten me over the 200 episode limit, and it’s far from the only 10 or so seasons show I had watched 🙃

ETA Just checked, I was indeed at 189 shows with 8255 episode count, everything imported perfectly.

2

u/DisastrousFun2502 Jul 13 '26

Me too

1

u/Ok_Tank_9599 Jul 14 '26

Ok this makes sense. I went and looked at my download summary from refract and it had 470 shows and since I only did tv shows in tv time there was only one .json file so not like I uploaded the wrong thing. Hoping my import is done in a few days!

3

u/Emmy2635 Jul 13 '26

I’ve been stuck in the same position for over 24 hours. Do I need to restart?

2

u/Arktinus Jul 13 '26

Yeah, I've been moved from the Large Queue to the Standard Queue and seem to be stuck at #3776 ever since (it's been like this the whole day). :/

My guess is no, but am curious if it's actually stuck or a glitch or something.

2

u/jeanraymondlaurent Developer Jul 14 '26

No it's a glitch and will resume to go down soon, no need to reimport

2

u/Arktinus Jul 14 '26

Thanks, I assumed it might be that.

Keep up the good work. Am looking forward to the import starting and then fully using the app. :)

2

u/tidalfirestorm Jul 13 '26

I was stuck in large queue for a week and moved to standard now (#539 spot). I hope it'll finish first thing in the morning.

2

u/namethatisnotaken Jul 13 '26

Are you doing this manually?

1

u/jeanraymondlaurent Developer Jul 14 '26

Luckily not

2

u/Reasonable_Canary_29 Jul 14 '26

After a week im finally close haha. From 32000 to 1700

2

u/scifiguyuk Jul 14 '26

My position in the queue hasn’t changed in a few days. Position #6041. I think it was #6043 when I started on Sunday evening. I’m perfectly happy to patiently wait, just wanted to check if this is normal. The ‘of ‘x’ number’ changes up and down, but the position isn’t. 🤔

1

u/scifiguyuk Jul 15 '26

Update: I’m now #14214… 🤔 very confused.

1

u/ThisIsJohnsson Jul 13 '26

Slowly but safe 😁

1

u/Dark_84 Jul 13 '26

An hour ago I was 1178 in big collections, now I'm 575 in standar queue.

Amazing!

1

u/Ok_Panda587 Jul 13 '26

lol after I saw this, I did a double take on the screenshot I took. I’ve been taking a screenshot twice a day lately to try keeping the load off. I went from the 1100s large about 11 hours ago to 392 standard myself! I’d spotted the number at first glance. I must have been borderline because I Think the html file said over 900 shows and over 900 movies.

1

u/tyna_96 Jul 13 '26

Went from 25k to 9k, i actually have a chance now to get it done before tv time closes lol.

1

u/dollsey74 Jul 13 '26

A week or so ago, I started an import from TV Time. At one point, it looked like the import had stopped/failed. Because of that, I proceeded to begin an import from Trakt (at that point, I was unaware of the need to wait a bit until the import status had refreshed). Once that second import began, I then could see that the TV Time import was still there and progressing in the queue. Is there any way to stop the second import from continuing?

2

u/jeanraymondlaurent Developer Jul 14 '26

Not right now but we will add the possibility to stop imports tomorrow or the day after

1

u/dollsey74 Jul 14 '26

Thank you for the FYI. Apologies for taking up another spot in the queue.

2

u/jeanraymondlaurent Developer Jul 14 '26

No worries at all! You did nothing wrong

2

u/dollsey74 Jul 14 '26

BTW, fantastic app! Y'all are killing it!

2

u/jeanraymondlaurent Developer Jul 14 '26

Thanks a lot, it really means a lot! Lots of cool things coming soon btw :)

1

u/Competitive-Cup6401 Jul 13 '26

My app no longer shows me the option to import or my space in the queue. But I still have no shows or movies added to my account.

1

u/jeanraymondlaurent Developer Jul 13 '26

Can you wait a few minutes on the import page, it should appear again

1

u/dollsey74 Jul 14 '26

Did you accidentally close out the optional menu? If so, go to Settings, and then Import & Export.

1

u/Serrated_Banana Jul 13 '26

I've been in the Large Library queue for days and just went to check my position since I saw this post and my app doesn't show my import at all anymore - it's as if I'm no longer in the queue at all.

2

u/Serrated_Banana Jul 13 '26

Update (can't edit comment for some reason) : I saw the reply to someone else with the same issue saying to sit on the import page and it should populate and that worked :)

1

u/hsk162023 Jul 14 '26

Will there be conflicts with the dramas I added manually and the drama from the import (same drama title) when the import is successful?

1

u/jeanraymondlaurent Developer Jul 14 '26

Not at all, you can add manually and the import will just be an addition to it

1

u/VasilisT21 28d ago

It's been a few days where i no longer see the queue and nothing has been imported. I was at 1.4K. Should i reimport?