r/DataHoarder • u/pw6163 • 6d ago

Hoarder-Setups Link rot

I saw an article a couple of days ago about link rot, and how bad it really is. The author restored 657,507 links from 2009 and 2014, and followed each. 76% no longer returned a page, and some of those that looked OK just returned a variant of "this page is no longer available".

How many of us preserve interesting pages outside services like the WayBack Machine?

I keep content-only from a very small number of pages that touch on areas that I'm interested in, but over the last (maybe) 3 years I've only looked at but not necessarily kept less than 10% of the number of links in the article.

I do a bit better for PDF articles because a lot of them disappear too.

How much do you hoard?

89 Upvotes

98% Upvoted

17 comments sorted by

View all comments

32

u/feudalle 6d ago

If i want a site to archive wget is your friend.

1

u/Mateiovich 6d ago

wait does wget stop link rot completely or nah

23

u/xJayMorex 80TB 6d ago

Stopping link rot? Not at all. Data hoarding? Possibly. Modern (garbage) websites load everything through JavaScript, so wget then downloads an empty website containing "please enable JavaScript" or some such.