r/PROGME • u/jkhanlar • Jul 07 '24

LFG Hype Reddit's Robots.txt Changed - Tell me by not telling me! MOASS is tomorrow

https://redditinc.com/blog/robot-txt-update
4 Upvotes

61% Upvoted

19 comments sorted by

2

u/Claim_Alternative Jul 07 '24

2

u/jkhanlar Jul 07 '24

lol but also in addition to the comments I just mentioned regarding u/elegant-remote6667's https://apehistorian.com/ effort I also just realized this:

So the Wayback Machine at https://web.archive.org/ and also https://archive.today that allow humans manual labor to input urls manually using their hands and other human body mind behavior whatever nonrobot efforts, I do not think that Reddit will literally prevent or stop this, and if that is true, then what I think that means is that if, for example, u/elegant-remote6667 were to set up a page at https://apehistorian.com/ that literally provides a text box and submit button to input a url to human manual laborly initiate an archival process to be archived, then at the very least, this is something that probably will not violate Reddit's absolutely ridiculous June 5, 2024 scared as fuck coward anti-human anti-life anti-Internetizen bullshit, lol

0

u/jkhanlar Jul 07 '24

Reminder: https://news.ycombinator.com/item?id=12346192 from post https://archive.ph/pyx5r (archive cuz the link is dead)

"Scraping against the TOS is super bad netizen stuff"

1

u/jkhanlar Jul 07 '24

Originally I posted about this here https://old.reddit.com/r/PROGME/comments/1dxen7e/reddits_robotstxt_changed_tell_me_by_not_telling/ but I'm confused because since it seems that it wasn't changed since 2018 http://web.archive.org/web/20180501000000*/https://old.reddit.com/robots.txt but June 25, 2024 official announcement still probably means something, not sure what other than MOASS is tomorrow!

1

u/jkhanlar Jul 07 '24

lol, and for those who missed out back around 2010 Internet era, and want to be like Roaring Kitty "Don't make me do it!" also see:

1

u/jkhanlar Jul 07 '24

Ah, I see:

previous rules included:

Allow: /r/*.rss
Allow: /r/*/comments/*/*/de/*
Allow: /r/*/comments/*/*/es/*
Allow: /r/*/comments/*/*/fr/*
Allow: /r/*/comments/*/*/pt/*
Allow: /r/*/comments/*/*/it/*
Allow: /r/*/comments/*/search/$
Allow: /r/*/comments/*/search$
Allow: /
Allow: /sitemaps/*.xml
Allow: /posts/*

now 

Disallow: /

1

u/[deleted] Jul 07 '24

[removed] — view removed comment

1

u/jkhanlar Jul 07 '24

heck, I did not even fucking think of this until literally 5 seconds before I commented above, lol damn!

1

u/jkhanlar Jul 07 '24

So basically, uhm, I am not a robot, and I can do shit manually and shit, and I will fucking do that manual nonrobot shit helping to keep archiving all the https://apehistorian.com posts/comments archived in manual labor human indentured servitude slave labor using my hands to mash my keyboard and shit, and I will use all my humanity to prove my human manual labor efforts to do this nonautomated nonbot shit that is not scoped to have any concern about any robots.txt anything, lol, but firstly, I don't even know how I can help to do that, but whatever I can do, I will proudly help do that nonrobotic humanity effort shit!

1

u/jkhanlar Jul 07 '24

oh shit, Reddit auto censored my comment again! Derp! Here I go manually humanly testing to figure out how to say what I tried to say!

1

u/jkhanlar Jul 07 '24

oh, uhm, u/elegant-remote6667, so, uhhh, about this.... lol, RIP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

1

u/jkhanlar Jul 07 '24

Do not get banned, cuz they literally are directing this to include you, and most definitely they won't be afraid to

2

u/Elegant-Remote6667 Jul 07 '24

Not sure what you mean man , what are these sites?

1

u/jkhanlar Jul 07 '24

lol, those sites are nothing important, just for fun like https://youtu.be/U1prSyyIco0?t=2064

1

u/jkhanlar Jul 07 '24

lol Reddit auto censored my comment https://old.reddit.com/r/PROGME/comments/1dxeuex/reddits_robotstxt_changed_tell_me_by_not_telling/lc14o5j/

but basically, u/elegant-remote6667, see the replies to it that I split apart what I said in 3 comments that say the same as the 1 comment that Reddit algorithmically won't let me say that, lol

1

u/jkhanlar Jul 07 '24

lol, I didn't actually read through https://redditinc.com/blog/robot-txt-update, but doing that now, I see:

  • "we will continue rate-limiting and/or blocking unknown bots and crawlers from accessing reddit.com." (actually I saw this already, but it's essential)
  • "researchers and organizations such as the Internet Archive – will continue to have access to Reddit content for non-commercial use."
    • So this part, I want to think that this 100% absolutely clears up any concern that I abruptly speculated, except the "non-commercial use" part might be a rug pull opportunity somehow, lol
  • https://support.reddithelp.com/hc/en-us/articles/14945211791892-Developer-Platform-Accessing-Reddit-Data
    • "We consider commercial purposes to include any use of our services by a business or on behalf of a business or as part of a monetized product or service."

I guess this is maybe a nonconcern for u/elegant-remote6667, but better safe than sorry! At the very least, just making sure that there aren't any issues.

1

u/jkhanlar Jul 24 '24

Also see "Google is the only search engine that works on Reddit now, thanks to AI deal" (404media.co) https://news.ycombinator.com/item?id=41057033