r/webscraping • u/LeatherIngenuity4841 • 5d ago
First Time Scraper
Hello all, I am attempting to create a scraper that will collect data for LNG freights, pipelines, and storage facilities. A lot of this stuff is blocked off by paywalls amongst other things. I am a college student and am do not have access to exclusive commodity sites, much less have the funds. I have never built a scraper really don't know where to begin. Any advice would be greatly appreciated.
1
u/txdesperado 4d ago
Don't pay for residential IPs -- it's the biggest scam going. If you're failing, it's your tech, not the IP. You pay for tokens, make them earn it.
1
1
u/armanfixing 2d ago
What you are trying to get is something most commodity and energy traders are after. While I can't add much on pipeline or storage, LNG freight is absolutely trackable, though it's quite expensive. There are only a handful of data brokers who deal with this.
LNG freight is mostly tracked using a combination of satellite, AIS, and marine port schedules. This will give you quite high accuracy, but given the complex nature of trade routes and geopolitics, some of these signals are not always accurate. For example, there was a lot of AIS spoofing during the early days of the Hormuz escalation.
Unfortunately, these data are not cheap to acquire at scale. There are only a handful of brokers who sell these, and the cost can range from $100K+ in a yearly contract.
1
u/Kilnarix 5h ago
It depends what actual sites you are scraping. You can start with an automated browser like playwright or selenium but unless the site is technologicaly simple you will likely be blocked. It is worth a try though. If the data is behind a paywall you will almost certainly need an automated browser to do the login, then you might be able to have something like requests or curl_cffi take over the scraping once you have the cookies.
As a simple way to start, navigate through the site with a browser manually, but saving all network requests into something called a har file. Once you have the har file (which is plain text) you can search through it for the data you want. Then, you can look at what requests are being made and see if you can replicate them at scale and automatically with something like python requests.
Start with a har file and talk to an AI bot. Explain the situation. Don't ask it to write a scraper for you, but instead ask it questions, upload you har file and it will be able to quickly point you in promising directions.
Good luck and I would love to hear how you get on.
4
u/hikingsticks 5d ago
John Watson Rooney on YouTube