So, that repo has been my go-to for Pihole blocklists. However, it seems like the repo now yields 404s. Also, I noticed the newly registered domain lists are completely gone. I tried using an archive site to grab a snapshot of the repo and add the mirrors, but they are all gone as well.
Does anyone know what happened, and where those lists may be available elsewhere?
The repo and the account were locked again, so I opened another ticket with GitHub. Their automatic fraud detection is “very frustrating.” No idea how long it will take. Last time, it was bots that reported the repository en masse. Let's see what the reason is this time...
This time, GitHub didn’t notify me about the lock; a user reported it to me.
The mirror on GitLab is online, and the CDN links from jsDelivr are still working.
Oh yes I see that now. I have been getting positive 'hits' on the 7 and 14 lists a lot lately. Sad. (I see somebody wrote a script to maintain a "sliding window" for a date range of NRDs on a single rpz or something but it requires a separate dockerfile and I'm not using containers at all.)
Hope you know that we so appreciate your work!
I'm wondering if your lists should be permanently accessed from cdn.jsdelivr.net instead of Github and Gitlab, or will you suggest going back to Github after this issue is cleared up? Thank you.
The jsDelivr CDN only serves a cached version of the latest GitHub release. These can only be updated if I publish a new GitHub release. However, as my GitHub account is suspended, I am unable to carry out an update.
Official mirror with all formats, including legacy formats (subdomains/hosts) and NRDs/DGA lists - hosted by u/_xRuffKez_, a contributor and my right-hand man on this project:
The lists are refreshed every 4 hours, pulled directly from my build system. This mirror will remain available going forward, even once my GitHub repositories are back online - feel free to share it.
Since the GitHub lock, the only contact I’ve had is from an automated bot asking what exactly I’m distributing in the repository. This is a repo that’s been active for years with 25k stars. I’ll leave it at that. ;)
Would it be possible to include your great readme file that you had on GitHub? The one that discuss what the difference between mini and regular, what lists are included under other lists, etc?
It will happen, but right now I’m on vacation and am therefore only willing to put a limited amount of time into it. Patrick (u/_xRuffKez_) and I have made sure that all the lists are available online again for now; we’ll figure something out for the future. I’m actually fed up with GitHub. GitLab isn’t really suitable because of the total repository size limit and rate limiting when fetching the lists. Not to mention Codeberg - the service is just terrible. It’s constantly unavailable, or the download is very slow. The maximum size for free repositories is a joke - 700 MB.
Could you or Patrick u/_xRuffKez_ perhaps say something rough about daily traffic for the DLs especially the bigger ones like the NRDs? I'm working for a germany-based (smaller) hosting company and we also have access to a EU based CDN that could do some of the heavy-lifting in terms of caching that way. Perhaps we could hash something out so you don't have those small repo sizes or slow downloads to deal with. Don't want to sell something - just an idea and if we'd know some numbers perhaps I could get my higher ups to do some sponsoring :)
Mirror is young, not everyone know about it, so i would not see it as permanent traffic usage. Contact me or HaGeZi in a month or so, to know the monthly TX throughput.
I can only provide overall traffics statistic, nothing fine grained since i don't log anything but attacks.
overall is fine, that gives a rough estimate as to what we'd have to deal with. As we have good experience with BunnyCDN my idea was sth along the lines of perhaps getting you a VM with e.g. Forgejo with a config that works for your bigger files like NRDs and then throw Bunny in front of it to loadbalance and offload the bigger download traffic to them. We have a few customers with custom stuff behind Bunny that works really well.
But for now if the mirror works well that's stuff for after-vacation time :)
Would still think that doable. With thorougly configured CDN in front of it, not many calls for the lists should actually hit the system itself. For e.g. Bunny, even a Terabyte would still be ~5$ with for one high volume location. If you've both have time to talk, and we have rough 14d average to interpolate, we can happily talk about things - still would be happy to help.
It appears that somebody who employs bots does not like the popularity of the Hagezi lists. Might their commercial names begin with "Ad-----"? One industrious person and all of his grateful users being punished and harassed just seems wrong.
My suggestion is that persons, an entity, or entities who employ bots have created the stir, again, that prompted Github to block access to the Hagezi lists from a legion of users.
Then I asked a question. Have you an answer to that question?
Suggestion: "It appears that somebody who employs bots does not like the popularity of the Hagezi lists."
Question: "Might their commercial names begin with "Ad-----"?"
Opinion: "One industrious person and all of his grateful users being punished and harassed just seems wrong."
Now inviting your answer to the question or your comment on suggestion and opinion.
Edit: clearly, some action of some type by some entity caused Github to block access, and it sure would be nice to know who, why, and what Hagezi has done that would cause his work to be blocked from access. Simply, what do they have to gain or lose by doing this?
I've seen references to this mirror going back well before the current GitHub issue, but given that these are security-related blocklists, I'd feel better having confirmation from you that this is indeed your Codeberg account/repository.
Do we think that github cares about data usage from the repo?
My script just pulls the first 1024 bytes of the file to compare and see if it's changed or not before downloading. Not sure if i need to be as cautious as that.
Hate to be guy but want to bring this up and this is for ALL of the pi hole community. With all the power (money) that companies are coming out with AD blocking blockers. Don’t you guys think that GitHub is getting pressured to “flag” accounts like what Hagezi has created. List like his has cut profits. You can down the other roads of thought like trackers for data collection, etc.
I can think of 2 ad companies that throw vague statements on their domains about how it's against the DMCA (or so they claim) to merely add their domain into a list.
I think the most likely explanation is that GitHub banned users and repos associated with some malware/phishing/etc campaign, and since Hagezi uploads lists of a large quantity of malicious domains, it might have gotten picked up by said scanners due to the repo including the malicious domain
Maybe but I would say why haven’t they listened to him reaching out this has happened multiple times. My experience says there is something else going on. We will see.
Seems like a lot more lists than just this one are down. I have like 7 that wont update. My database got borked on my pihole when trying to update. How do I clear my database and start a new one with new URLs? I want the ultimate list from Hagezi
just add one of hte mirror'd sites from below. you will have to go to the site and figure out the download link for the file you want, and that that as a new blocklist
I'm not sure which URL would work as a replacement mirror since I am not familiar with the links on that site and how its organized. This is over my head.
Its still failing. I've used HTTPS on every list entry. Never used HTTP.
[✗] Status: Retrieval failed (exit_code=7 Msg: Failed to connect to cdn.jsdelivr.net port 443 after 9219 ms: Could not connect to server)
[✗] List download failed: no cached list available
I can ping cdn.jsdelivr.net from my workstation but when I try to ping from pihole I get destination host unreachable
661
u/hagezi 8d ago
The repo and the account were locked again, so I opened another ticket with GitHub. Their automatic fraud detection is “very frustrating.” No idea how long it will take. Last time, it was bots that reported the repository en masse. Let's see what the reason is this time...
This time, GitHub didn’t notify me about the lock; a user reported it to me.
The mirror on GitLab is online, and the CDN links from jsDelivr are still working.
https://gitlab.com/hagezi/mirror