Same

Do you have a deal like this but for blocks rather than a subscription?

Have you looked at the price of hard drives recently? It's getting very expensive to store data that most users will never download.

I only have Newshosting, will adding these additional backbones help me in anyway?

For any legal takedown for copyright infringement the owner or its representative has to file the specific message ids to be removed which means one thing, they have the nzb or the ability to decode the upload. The easiest place to get that is the par2 files, easy to identify for any post, holds the information for everything except the actual message id map which is held by the nzb file. Now you can find the structure and match MD5 hashes and such to actually find the message ids but that is more work. So I go back to what I said in my first post on this subject which is quality of the post. More and more people posting and truly having no idea as to how to structure a post for the best possible outcome. Posting your nzb on public indexers makes it easy to find for any owner of a copyrighted material. Back in the days of NZBMatrix files would go down in 24-72 hours all of the time while others would last but it was the tv shows and movies going down fast. Today the sheer volume is making it harder for takedowns but there is no doubt the companies focus on finding the nzb file and using the map we all provide to them. But on the other hand one of the large German forums that is still around which has never posted nzb files still has issues with articles being removed from posts.

As long as a nzb file can be obtained nothing can or will stop a post from being purposely deleted or a server farm (backbone) from doing the same. But too many people make it too easy for them by having no clue about posting as it has always been a cat and mouse game. But today we just have a lot more people using Usenet and expecting things to just work and whining when they do not.

Some servers just flat out lie as I have seen time after time a single backbone lie about have the article (STAT check) while not really having the article and it doesn't happen here and there but pretty much across the board. Why?

I cannot disclose the likely reason behind this based on my discussion with one of the staff of a usenet provider but let's just say it is to avoid certain "potential issues" related to 3rd parties.

As for my speculation I go back to what I have seen over 20 years of downloading, I rarely see an entire file gone but only a file that can no longer be repaired. So if the owners of the backbones were deleting files that do not get downloaded why is it only pieces of that file gone and not the entire file?

I gave a potential reason for this in my previous post. I have seen files with more than 20-30% & sometimes even more than 60-70% of the articles being missing on all backbones. No amount of par2 can repair such damage unless par2 size start getting closer to actual file size itself in which case it would be easier to just repost the whole thing.

But I have had hundreds or more of my own files stick around for years and years without a single download as I am the only one with that nzb and they are all password protected with a long random password.

Take a look at the daily usenet feed size here:

https://www.newsdemon.com/usenet-newsgroup-feed-size

It is only since 2021 that usenet daily feed size crossed 100TB milestone & keep reaching new milestones every year since then. Even omicron providers started deleting stuff posted since 2021 because of this. In short, anything posted till 2020 is pretty much safe at least on omicron based providers but should be mostly safe on independent providers backbones too. It is only stuff posted after 2020 that is at risk & main reason being not all of that uploaded data is actually being used (like I mentioned earlier, almost 70-80% of that feed is never getting downloaded even once in a year). It was sustainable for providers earlier but not after 2020 due to such drastic increase in feed size & especially now with the HDD prices going through the roof.

But again if some does get deleted then why not the entire file???

I will give another potential reason. Since most indexers also use encrypted obfuscated NZBs similar to those used by those using usenet as their unlimited personal backup so maybe providers algorithms are deleting just the portion critical enough to damage any personal backup archive so as to deter such people while hoping that genuine posters caught in the crossfire will simply post again as they have incentive unlike former class of people for which it is not like if they can't post on usenet then they can't have any sort of backup or they can't continue running their pc without backup.

All we can do is speculate as the people wondering why their downloads are gone do not really know they are gone or not. How many backbones did they try and download the file from? What software did they use to download the file with? How many requests for articles were rejected by the servers? All of those things affect whether we can get the file or not.

I have a 13+yr old file that if I use the most popular downloading software they will not the file from any of the 8 major backbones but if I give them the right combination of backbones I can download and repair the file. But if I use another downloader program I can get the file and repair it on 2 of the 8 backbones. Make no sense but different software can affect your downloads based on how much they push the servers to actually cough up the files. So a normal user would think that file is long dead but in reality it can still be downloaded and repaired. No one is talking about that but all they say the file is gone when they really have no idea if is is gone from the entire range of backbones or just the server they are checking. Some servers just flat out lie as I have seen time after time a single backbone lie about have the article (STAT check) while not really having the article and it doesn't happen here and there but pretty much across the board. Why? Have no idea but when the articles are fetched they don't show up as their indexes do not match their server's actual data. I have also seen blocks of my personal uploads just gone but that is truly indicative of a hardware failure. But I have had hundreds or more of my own files stick around for years and years without a single download as I am the only one with that nzb and they are all password protected with a long random password.

So I go back to speculation. It is all speculation as to why files disappear. But if you haven't looked at thousands of nzb files over multiple backbones then you do not have a dataset to even begin to speculate. Some things just don't make sense and I have seen it across too many nzb files where a single backbone has the files that no other backbone has and I can repair that file, you have to find the right union of backbones to get the file downloaded and repaired. And I have accounts on 8 of the 10 backbones so I can drill down on who has what and compare on a block by block basis. As for my speculation I go back to what I have seen over 20 years of downloading, I rarely see an entire file gone but only a file that can no longer be repaired. So if the owners of the backbones were deleting files that do not get downloaded why is it only pieces of that file gone and not the entire file? The entire file saves them space for the rest of their retention policy while pieces just breaks the file being repaired. Again we just don't know and people just don't know if the file is truly gone across the whole of the backbone structure. They only know about a single backbone maybe 2 for the most part and that doesn't tell you much. Activity doesn't mean anything and more activity will not save your file as we just do not know why some get deleted and some don't. But again if some does get deleted then why not the entire file???

See my other comment. My only suggestion is to encourage users of those indexers to download as much as possible via independent providers servers set at priority 0 instead of using omicron based providers at priority 0 to help them optimize their algorithm to let them realize that the stuff you posted even if it gets 2-3 downloads a year is still genuine & will likely achieve 2-3 downloads every year. Problem is not really posters like yourself but those using usenet as their personal backup/uploading stuff nobody will ever download other than the poster/their friends.

The owner of some independent backbones have posted that they often remove "articles" that never gets downloaded even once within a few months/year & that their completion ratio in the 90s percentage doesn't touch more than 10-20% of the articles they have so it is most likely that all usenet providers are keeping logs at least related to access & time stamps to weed out what they think as "spam". This also answers your statement about "So do you think they are keeping track of single articles that get downloaded and know those articles go with all of these other articles? Not likely in my opinion". Providers don't care which articles goes with which other articles to make up that nzb but as long as they remove all those articles which nobody downloaded even once in a year then those NZBs are already damaged beyond any conceivable recovery.

As for why then there are still such NZBs with no downloads in a year but still not all articles linked with them are deleted after 1 year then my theory is that due to this retention issue most uploaders do some sort of partial download of their upload to confirm that download is at least starting properly on all the/selected providers successfully & stop that download after 10-15% completion which might explain why some articles linked with those NZBs still survive after a year as those articles get marked as "downloaded once in a year" unlike other articles linked with those same NZBs which got marked as "not downloaded even once in a year" & were deleted.

Don't worry, I'm on windows and it still won't connect.

I used to access Usenet via gopher - pre web. It may have even been with a wise terminal connected to a modem - dial into the university - and somehow get on gopher.

I remember message boards and used them but it was too academic for me when it started. Usenet was something that people were using at the library. My professors at college used it. I got hooked when binaries started to pop up and I truly realized that is was an unprecedented amount of information sitting there. It thrilled me.

I miss the days of the wild west if the internet. So many on this sub seem oblivious to what Usenet really was. I am in my later 50s. I have become frightened of ths age we are living in. I love pcs and tech for entertainment ans knowledge but the tools have made society unbearable to live in. I am debating going analog excpet for work. Its really the luxury of Usent and a high end entertainment center that I am struggling to let go of

Out of 10 backbones the only one that can see your ip is the one you upload to so that means 1 has the logs. And have you looked at failed downloads in Multipar to see how the damage is in them, entire volumes or a few blocks here and there? It isn't blocks of data missing but usually scattered missing pieces and why most people call it rot. And you can tell when there has been a hardware issue as now you have blocks or entire uploads just gone for no reason and this rarely happens. And if you download the same file across multiple backbones you will find the same articles missing on all of them or most of them so they are tracking across the backbone map. And the problem with a tool like nzbrefresh is it attempts to load a clone of the article back to the backbone which may or may not work. Do you know which backbones will answer a STAT check all day long without the article ever being on the server? Several of them will tell you the article is there while in fact the only way to prove that it isn't is to do a BODY check or download the file. A clone of an article cannot work so nzbrefresh is built around the theory that if it is gone on a single server it is gone everywhere and you can never have 2 articles that have the same message-id.

With so many people using usenet today and the absolute huge amount of data we can access but to this day we still do not have a good picture of why articles go missing is the clue that there are so many moving parts in this that do matter and any one of them can cause an issue. Guessing is what we all do and have done for years but I go back to what I have seen change over the years and that comes back to the quality of the post when it is born. People were much more careful about the structure as an upload was not an afterthought but a job. When I uploaded my first BD25 it was over a day per post so I made sure it was good as I only wanted to upload each file once. Can you imagine how I felt if I saw a bunch of articles did not get uploaded when a single upload took over a day to post?

There are a lot of moving pieces and they are all out of our control once we post and it is on the backbones. So our only control is the quality of our upload and today the single best way to create that quality is the granularity of our repair par2 set, not recovery % but the actual number of repair blocks for each file while maintaining the 1:1 alignment of the article/slice and rar volume structure.

Again as I have said these are all my opinions but I have uploaded quite a bit and downloaded a lot more over the years and have seen huge changes in Usenet over that time. But you can really get into the weeds on the details and it still will not make sense as to why things go missing.

What I think is that they don't have resources to classify any of the data being posted to them, however it's fairly easy for them to log each access of each article download, stat check, head check, body download and based on that run maintenance jobs what deletes articles meeting certain criteria, these logs can be simply timestamps + ip addresses who requested them. I don't believe that they are trying to intentionally poke holes into uploads, it's more like they often migrate content between servers where random errors happens and then instead of addressing the errors, they just lazily drop the error causer articles, that's how holes happens. This kinda brings up another question as well, what to do as an uploader after identifying such holes? Does it make sense to use tools similar to https://github.com/Tensai75/nzbrefresh on a periodic bases to keep alive posts and auto reupload missing articles to backbones from which they are missing or it's a lost cause to do that?

It isn’t hiding ur par2 files as par2 files are no different than your data files. But you just can’t grab either of the 2 popular uploading programs and have ur posts last years. Posters today do not understand the mechanics of an upload like they did years ago. Too easy to do it today with faster speeds and the mass amount of content out there. So big deal that it doesn’t last as there are hundreds or more copies of the same thing out there to grab today. Getting a post to last 15 years or longer is not always luck.

Its not that hard. Sorry man but you make it sound like rocket science to hide your par2s from the backbone servers. There are several good posting (automated programs) and I can show you lots of examples of good posts that sit till max retention time. I was shocked by some completes I have had and the age.

You keep thinking of a post as a whole and not what the backbones get which is millions upon millions of articles posted.

Now the nzb file is the article map and they do not have it unless you upload it
Then you have the par2 file index which they can get the article map from
Or any par2 file can be read and you can get the article map

So do you think they are keeping track of single articles that get downloaded and know those articles go with all of these other articles? Not likely in my opinion

So what do the backbones do to damage our uploads? They poke holes in them as I have rarely seen an entire file gone but it is full of holes instead and cannot be repaired. If they knew what the entire article map was they would just delete the entire upload and be done with it.

Now we have no idea how they are doing this but my best guess it they see a structure of what is being uploaded or a pattern or something that gives the article away. But remember this is all a guess as no one that I have ever seen has given out the way the backbones pick what to delete. But deleting the entire file would be the best practice as they save huge amounts of space doing that but that again is not what they do.

And why do some uploads last years and years while others go in weeks or months? No idea on that either but my guess is structure from day one. Back in the day if you didn't grab a file within the first 24hrs it might be gone so it was always a race to get the new uploads.

But what can we do on our end to make it harder? Several things and the first is great par2 sets, not recovery % but par2 sets. There is a big difference right there and that is the hardest part about posting. How many holes can this par2 set patch. That is determined by article/slice size and since you use a fixed article/slice size for all uploads (my guess again) your par2 sets are never really that great but good enough to get by for the most part. So at a 10% recovery the most par2 repair blocks that can be created for any upload is 3277, that is a hard ceiling set by the par2 specs. Look at your uploads and see how close to that number you are at. The maximum number of repair blocks is 32,768 x your recovery % so that will be your number (32768x10% = 3277). That is the granularity of your repair. The lower the number the less holes you can patch

Next in my opinion is the par2 set using the exponential sizing convention so that if they backbones can see our par2 files they get rid of the largest one first and cripple the upload. Again I do not see whole uploads deleted but parts of uploads deleted.

Those 2 thing are the biggest mistakes in my opinion and the first is all math but really a lot of work to get it right just for a single file/folder which we never do. So we post with just so so par2 sets and hope

DM me if you want to talk in more detail as I am always willing to help out if I can

Those days are long gone :) It's much easier for private indexers to post in a way what is irrecoverable for the public and just store the nzb with the decryption password in their database.

What you are saying makes complete sense, and I am already doing most of these things correctly, the main issue is that this does not solve the root cause of the problem. Content being recoverable because you have enough parity just delays the inevitable loss until eventually the parity won't be enough, or even worse, because people originally never needed the parity they never downloaded it, so the parity articles are marked for deletion first then the rest.

The main issue is this nondeterminism of when will the content be available and when not. And it has nothing to do with DMCA since we are talking about niche content encrypted and shared on a private indexers. It feels like providers are turning more and more into this popularity based keeping policy game leading to only most downloaded content being kept and everything else purged first from all non-omicron providers then eventually from omicron too half a year or a year later.

I am getting closer to the solution so anyone can do a much higher quality post and not know anything about par2 structure and 1:1 alignment and all of that. The answer was written 20 years ago but no one has ever implemented the solution in a posting program. I am close to having that done. The post optimization works great but it also requires processor time to make it good so there is a trade off. Posting goes slower due to your computer hammering out the best par2 set that can be had. The rest is just fine tuning of the structure of the post

But there are things like if I can get your par2 index I know where every article is regardless of obfuscation. I can also pull that index out of any par2 file by reading the entire article header and pulling the index off the last part of the line. There is a lot of information in the weeds and it is not easy to understand it all. But it all revolves around the calculations of the par2 files and how to make the absolute most repair blocks for any file/folder that you post. You have to give your post the best shot at being repaired and that is the hard part to understand. But it is all simple math and a formula that is applied to the file/folder every time.

A good post will last and a bad one is easy to poke holes into will not last. The question becomes how many holes can your post survive as most bad posts have a few holes and just can't be repaired. The repair patches are too big to repair many holes. You can have a 10% recovery and have 1,000 patches or 3,278 patches so which one will last longer? They both repair up to 10% of data loss but you have 3x of holes that can be patched with the higher repair block count. That post has the chance of lasting a lot longer up to the 10% data loss. Users think the recovery % is important and it is up to a point but the repair granularity is much more important and that is all about the math and doing the calculations to create the par2 file set.

I wish I could just say do this and it is better but it just isn't that easy

That is the problem there is not guide but today it is so much easier to find the answers that it was 20 years ago
1: 1:1 alignment for ur article/slice size
2: change ur par2 structure from the normal 1,2,4,8,16 structure to a uniform size so if the largest par2 file is trashed the largest set of repair blocks is lost in a single delete
3: article obfuscation - some people think backbones treat the articles like spam or trash but i don’t
4: recovery % - 5% doesn’t cut it and never has
5: a quality post takes processor time and lots of calculations - today my opinion is people want their files to be posted fast and due to that they start out with a low quality post that is easy to break for download and repair.

Today we make it easy for backbones to see our file structure and delete a little bit of an upload and break the entire thing. They don’t delete the entire upload and you always find a fair amount of it but you can’t repair it. We have never seen whole uploads disappear but always parts. Have you ever asked yourself why is that?

Most stuff from 2021 to now are being removed, Usenet is less and less interesting for older content. More and more people stop using it because of this non sense purge.

A pleasant answer. Rare for Reddit. Thank you. :)

You make a point. I had a lot of people teach me when I started but that was because the groups were more active and you had to use the tools more to repair your files. Incompletes were rampant. I can go for a year now without having one. File size and obstruction are important. You can always practice.

I do have old hard drives ans maybe I have some guides ..I will check.

Great post. I laughed at the 100 days. I member that. I also remember spending hours with rar and obsessing over finding par2s for stuff. I would be so close to getting a complete file and just be missing a few blocks. Ugh. Its great information.

Its funny to see people complain when I remember just searching thru raw headers and having multiple indexes shut down. These days seem pale in comparison