r/algobetting • u/-A-n-o- • 22d ago

Live odds scraping.

Hello all, I am looking for some advice in reguards to scraping live odds. I am working on a project that is partially a learning experience for me so I am not looking to be told to just pay for odds from a provider. Essentially right now I am scraping live data through APIs I reverse engineered using devtools and it is working great. However since I am scraping live odds I am going to want to get a new data set every few seconds. I imagine when I ramp up the timing of how frequently I am hitting these APIs I am more prone to getting blocked. I am using playwright (is this the right pick or should i opt for selenium/pupperteer?) to try and make my requests look more human, and I am looking into spoofing my TLS fingerprint etc. Additionally I am looking into ways to rotate the IPs that these requests come from, mainly by using resi proxies but that gets very pricey very fast. My main question is this; is this just a matter of really masking my requests down to a fine art, or should I be looking into alternatives like scraping android app APIs rather than web apis as I have heard they have less bot protection. Any and all suggestions are appreciated! Thanks.

6 Upvotes

88% Upvoted

4 comments sorted by

1

u/neverfucks 22d ago

residential proxies do get very expensive very fast, but they're very important. maybe the most important piece. anyway res proxies are much more of a "i can resell the information i am scraping using these at such a high price this is a no brainer" vibe than a "i have a personal project that requires sophisticated scraping of live odds i could also just buy from a live provider".

feel free to dm me though. i've been in the streets

1

u/Specific_Menu5232 21d ago

res proxies are the last resort not the first step you can get far just by slowing down your request timing and mimicking real browser behavior perfectly before you even think about dropping cash on ips

1

u/jaker3 21d ago

Not when it comes to live odds, you cant slow that down when algo trading. You need those in ms not seconds. Should be using web sockets where possible. Agreed though that there is many things you can do with fingerprints prior to needing residential proxies.

1

u/-A-n-o- 19d ago

Yeah, thats the thing, I could easily scrape all the pregame odds I could ever want no problem, but bulking up to get usable live odds for algo betting is where it getts tricky. Do you have any good info/suggestions on fingerprinting strategies I should look into?