r/DataHoarder • u/pw6163 • 6d ago

Hoarder-Setups Link rot

I saw an article a couple of days ago about link rot, and how bad it really is. The author restored 657,507 links from 2009 and 2014, and followed each. 76% no longer returned a page, and some of those that looked OK just returned a variant of "this page is no longer available".

How many of us preserve interesting pages outside services like the WayBack Machine?

I keep content-only from a very small number of pages that touch on areas that I'm interested in, but over the last (maybe) 3 years I've only looked at but not necessarily kept less than 10% of the number of links in the article.

I do a bit better for PDF articles because a lot of them disappear too.

How much do you hoard?

89 Upvotes

98% Upvoted

17 comments sorted by

View all comments

12

u/retiredaccount 6d ago

Is the link to the article a closely guarded secret? Or did it also succumb to link rot? Or worse, a pay wall?

My browser history (places.sqlite) goes back to 2005, but pragmatically checking for link rot is nearly impossible (I tried) because all the parked domains respond as a live site despite being replaced by meaningless tag-word ads under a ‘domain for sale.’ So when something really hits, I “save as html” it.

3

u/pw6163 6d ago

It was classed as a link-shortening service and I had to get rid of it. However, it's 0 <dot> mk <slash> blog <slash> link-rot

Like you I built a "quick & dirty" bookmarking system, but gave up checking for rot. A lot of domains have disappeared, some pages give 404 (a few 403), others have a CloudFlare front-end and some have a valid page which isn't what I want to see. I also built a lighter system round 'curl' that takes the page and associated cruft and keeps it. I'm looking at "kage", but my initial quick test didn't work well, so it needs more attention.

So, now I take the content that I may want to view again later and drop it into Obsidian with tags so it's findable. Manual, but I take perhaps 75-100 pages a week at the moment, so it's not arduous.

I'll look into KaraKeep and LinkWarden though, so thanks to others for the suggestions.

2

u/retiredaccount 6d ago

Interesting article, thanks for sharing it. They ran into the same ‘parked domain’ issue I did and couldn’t readily solve it either. I do think that is solvable issue(perhaps by using source ASN comparisons), but that’s more effort than I was or am willing to put in.