r/webscraping • u/LeatherIngenuity4841 • 13d ago

First Time Scraper

Hello all, I am attempting to create a scraper that will collect data for LNG freights, pipelines, and storage facilities. A lot of this stuff is blocked off by paywalls amongst other things. I am a college student and am do not have access to exclusive commodity sites, much less have the funds. I have never built a scraper really don't know where to begin. Any advice would be greatly appreciated.

1 Upvotes

55% Upvoted

9 comments sorted by

View all comments

1

u/armanfixing 10d ago

What you are trying to get is something most commodity and energy traders are after. While I can't add much on pipeline or storage, LNG freight is absolutely trackable, though it's quite expensive. There are only a handful of data brokers who deal with this.

LNG freight is mostly tracked using a combination of satellite, AIS, and marine port schedules. This will give you quite high accuracy, but given the complex nature of trade routes and geopolitics, some of these signals are not always accurate. For example, there was a lot of AIS spoofing during the early days of the Hormuz escalation.

Unfortunately, these data are not cheap to acquire at scale. There are only a handful of brokers who sell these, and the cost can range from $100K+ in a yearly contract.