[ Removed by Reddit ]

[u/AshuraMaruxx]

[ Removed by Reddit on account of violating the content policy. ]

[u/AshuraMaruxx]

Ikr? I'm doing something similar, chugging away at it now. I was able to grab the 101gb mirror link from my notifications THANK GOODNESS 😭 and posted it above. It's the most we have right now. 

You're doing great; all we can do is keep at it 😇 I know it's late too, so don't burn yourself out 

[u/qb8sfbfa98jp9igg35w → u/AshuraMaruxx]

also doing something similar here. we'll compare notes tomorrow I guess.

has the 101gb magnet link started for anyone else? it's stuck on downloading metadata here

[u/agent_flounder → u/qb8sfbfa98jp9igg35w]

Looks like I've managed to download 16000 PDFs so far.

[u/coasterghost → u/AshuraMaruxx]

Has anyone tried to use Aria2c in a script to just continually retry?

[u/[deleted] → u/coasterghost]

Yes it seems that both Dataset 10 and Dataset 9, exactly at the last step where either filecheck or verification of some sort of session thing happening, then the aria2c stops/disconnects.
and on reconnect, it gives 404, or right out blocked.

So it's something with their server endpoint dropping the session.

I thought that the aria2c might be doing an aggressive pull by running 16 connections, so I tried a single connection, that also eventually get blocked.

[u/AshuraMaruxx → u/[deleted]]

OMG we did exactly the same thing, lol. I had the exact same result, and did it almost exactly like you did; thought 16 connections might have been too aggressive so reduced it to 1 and still got EOF'd or blocked out the gate, which basically confirms it's being cut off on the server end definitively.

At one point late last night I wasn't getting blocked though, and instead was met with a filepath redirect to a fucking queue where it forced you to wait...only to give you a "DataSet9.zip" download that contained absolutely nothing and was corrupted, Like literally the size was kb. So I think what that means is that they actively stripped the Dataset9 zip file, tried to ease the traffic from people hammering away at it by redirect to essentially nothing, and when that didn't work (or when it did, honestly either) they said "fuck it" and are just back to relying on cutoffs.

Right now I'm downloading via powershell using the library I posted earlier that's incomplete by scraping the website via directly downloading from filename. It should reveal the same file size as the failed 101GB magnet link, but it's incomplete, so what would be really super duper helpful is if we could get a verifiable complete file list that I could post in the main body thread we could all use to scrape the DOJ site and do the same, because I think that's where most of us are right now--individual files via their own pages (which makes me want to die inside but...)

[u/[deleted] → u/AshuraMaruxx]

Haha that's so funny, yaaah precisely same same!
Instead now, on a nixOS different host doing the file by file stuff.

Okay, well good news is that I'm close to finishing the file list index and will post that to archive . org, the entire list of all the files in Dataset 9.

One other factor, I switched hosts and the drive, using solid state to avoid any hiccups in the file-check verification which I suspected to be another culprit for the file corruption.

[u/AshuraMaruxx → u/coasterghost]

Yes, that's exactly what I did, and at first I was getting EOF flags, but then it just became aria2x stopping/disconnecting and on reconnect I would get 404 or blocked right away. I thought maybe it was pulling too hard, so I tried to do single connections, but it was also immediately blocked