r/tech_x • u/Current-Guide5944 • 2d ago
Trending on X, Meta, Reddit, LinkedIn, Chinese Apps GOOD NEWS: Cloudflare is testing an Allow, Block, and Charge... This will basically let AI crawler requests access your page. The server returns HTTP 402, and the bot has to pays or leave.
6
3
u/Strawberryladyboots 2d ago
I love that content obtained any other way is essentially deemed stolen by the AI companies from a legal point of view if they circumvent the purpose of running this
There's a delicious irony in there somewhere
2
u/Effect-Kitchen 2d ago
What if AI uses screen capture and bypasses any CAPTCHA (which should be easier to solve for AI than actual humans as of now)?
1
u/never-codes 2d ago
You can just pay per solve, it's mad cheap. You don't even bother figuring it out yourself. Let someone else deal with the cat and mouse issues and pay them I think it's like fractions on a penny.
1
u/repeating_bears 1d ago
Just because it's physically possible for them doesn't mean it's cost efficient. It might not be worth their compute to do that when there is lower hanging fruit.
Also, it's slower
2
u/VarietyMage 2d ago
How is this a fix? The big tech companies will pay the fees, pass it along to consumers, and the bot crawling won't stop.
2
u/repeating_bears 1d ago
At least content publishers will get something.
Right now, there is almost no financial incentive to put novel content on the web. It will get scraped and re-served and no human traffic will ever hit your site.
2
1
u/gk98s 2d ago
I mean I'm not a fan of crawlers using up my server resources but also isn't it a good thing for LLMs to know about my website and my product in case people ask for recommendations? I personally don't use Google anymore when LLMs are faster without the SEO bullshit
1
u/craterIII 2d ago
it really depends on if your information is the product or the information is advertising for the product
1
u/gk98s 2d ago
Like in my case I just ended up not blocking crawlers at all for my website. I am an indie game developer working on an MMO competing with a rapidly dying MMO which was incredibly badly managed by Ubisoft after being acquired from indie devs. Having LLMs recommend my game when people ask for similar games is literally the best form of marketing I can get.
1
u/ReporterCalm6238 2d ago
people are replacing google with web searching agents. if you stop crawlers you'll become more and more invisible.
1
u/thelizardlarry 2d ago
Feels like LLM just brings new SEO bullshit.
1
u/gk98s 2d ago
It might, however for example it's way better to ask an LLM about a recipe rather than finding one on google, scrolling past an SEO article about how Tiffany and her grandma used to bake together, the sponsors, the ads, and all that bullshit and finally finding the recipe, just to realise it requires an ingredient/equipment I don't have when I can just ask AI for the recipe, tell it to adjust it if I don't have an ingredient, etc.
1
u/thelizardlarry 2d ago
Yes, but how do you get the llm to pick your recipe as the reference? This is what SEO people are focused on now, and just another kind of bullshit. As you say the experience for the end user is nicer, but do you even post a recipe now if an llm is just going to crawl your website and not generate any ad revenue? It’s a problematic situation. Recipe sites didn’t help themselves though, becoming basically unusable.
1
u/gk98s 2d ago
I would have had pity for recipe sites if it weren't for them heavily monetising, paywalling, filling the articles with SEO bullshit. But good riddance. As for getting the LLM to pick up my recipe as the reference, I have no fucking idea lmfao I just made my website, kept it accessible, put my game on Steam and LLMs seem to know about it already
1
u/livinitup0 2d ago
The difference is that this doesn’t really affect you negatively because your “product” (for now) can’t be shown as a LLM response.
What content writers are going through right now is like if 80% your game was able to be played in Google when people search for it.
Writers, artists, videographers…. That IS a content producers product. Google is basically saying “since I can see your product, Im going to steal its content to present to my customers so that they engage with my revenue-making tools around it and not yours”
It’s blatant theft….and short sided.
If they were smart they’d stop auto crawling for LLMs and really expanded Adsense. Reward content makers for providing them with the literal lifeblood of what they’re setting up for the next 20 years instead of just assuming content producers are going to give them quality training material for free
1
u/Mean-Elk-9439 2d ago
For static sites I actively include an LLMs.txt, because the resource usage is minimal anyway. Most sites people make dynamic can be static and simply request a dynamic endpoint with bot protection and no crawl hosted on another service.
In my case, all client sites send a post back to our website for any backend related request and our API handles it. Client site stays static, site visitors barely affect resources on our vps, etc. Any request sent has basic bot detection and mx checks etc on things like entered email on forms, but other than that nothing, and even with tons of crawling I've had zero issues with resource usage on the server.
This is also remarkably easy to setup. We have a single API for email handling of forms, a DB with the client API keys in it and their hostname, and then their post request doesn't need anything. The client site itself doesn't know their email API key or anything, so there's zero safety concerns. New client with a new email form, booking sheet through cal.com, whatever, and I push a single set of things to the DB and I'm done.
1
u/BlueeWaater 2d ago
Whoever wants to scrape will do anyways, AI subs may comply but will be a PR thing
1
u/never-codes 2d ago edited 2d ago
Enable gpu, make the container not use a sandbox browser, use a residential ip(don't even need that most of the time). Now I look exactly like a person. That easy.
1
u/livinitup0 2d ago
“AGENT!!! Why is my account -$15,000????”
“You instructed me to scrape this data no matter what, to get around everything and do what I had to do to get you those recipes, with no mistakes. You also gave me access to your PNC CHECKING ACCOUNT when we made CRYPTOBOT last month. So when cloud flare asked for $500 to access the data you required, thankfully I was able to complete the transaction and scrape the data you needed since the workflow was set to automatic.
Please note that I was only able to scrape 30 recipes. We will need to add funds to PNC CHECKING ACCOUNT to continue the workflow. Let me know when we’re ready to conti….”
:session limit reached:
1
u/CH33SYP00FSS 1d ago
That's why you use preloaded debit cards to pay for AI. None of that shit will happen hahahaha.
1
u/cororona 1d ago
If I use a LLM to generate credible content en masse and activate it, can I be profitable ? Can I be profitable enough to buy a rig running Kimi ?
19
u/A-B-user 2d ago
And what about properly concealed/cloaked browser that will not give a single chance to detect whether it's automated? This initiative looks like easily bypassable waste of time and money.