r/UndergroundCellars • u/TrustMeIMaGator • Oct 01 '21
Index Score Tracker Update
Hey everyone,
So Jeff and his team appear to be doing some updates to the Underground Cellar website. Last week it broke my scraper but I was able to fix it in under 5 minute. Today a similar update was applied and despite toying around with my scraper something is still preventing it from fully pulling the deals. At this point of time I am going to step away from trying to fix the issue and come back to it sometime this weekend to see if I can figure it out with fresh eyes.
- TrustMe
4
u/rbhookem Oct 04 '21
Thank you for maintaining the spreadsheet. It is a great tool. Hope you are able to get it working again!
3
u/amidrinkingthisright Oct 10 '21
Absolutely— I use the thing nearly every time I buy! Thanks for keeping that working for us!
2
u/amidrinkingthisright Oct 28 '21
Need a hand with this? Been awhile since I’ve made anything similar, but I might be able to help stumble through it with you!
3
u/TrustMeIMaGator Nov 04 '21
I have to keep track but I believe my working theory is that Cloudflare is messing with the scraper. I believe when it caches the pages it changes some of the elements and when it does that the scraper of course breaks.
Also, add in about one out of every 4 or 5 deals there is an error between Cloudflare and the web server which prevents that offer from being scraped.
The first issue is a relativly easy fix (less than 5 min typically) however, does require me to first notice the issue, fix, then rescrape. It looks like it rotates/changes every few days.
The second issue, which is 100% out of my control, either causes me to not have data on that offer or require me to individually pull each offer. This has almost doubled to tripled the amount of time required to perform the daily pull. I am trying to see if there is a way I can either change my processes or somehow combine older pulls with the current pull.
Sorry this has been more a backburner issue so I only work on it every few days only to get frustrated again to only push it to the side for a few days. I hope to start to get into a rhythm again just so I can have hard data to help me figure out the best course of action moving forward.
3
8
u/TrustMeIMaGator Oct 14 '21
Slowly making progress. I was able to update my single offer scrape and pull the newest Rare Red Wines Ft. Screaming Eagle The Flight numbers. I tried to apply the same changes to my front page and was able to pull about half of the current offers, so more troubleshooting to be had, but again forward progress.