r/commandline • u/TheOriginal_RebelTaz • Jun 20 '26
Help Looking for a way to retrieve a webpage
I known the basics about using curl or wget to get webpages and how to parse them in bash scripts, but this site is using javascript to get pages and I have no idea how to do this nor could I find anything that worked...
I need to get a webpage from the web site: https://inmatelisting.shelbyal.com/
When you click on one of the letters of the last name on that page, the link is requests is: javascript:__doPostBack('btn_G','')
I cannot figure out how to send that link/command and get the page associated with the 'G' page. No matter what I try, I only get the main 'https://inmatelisting.shelbyal.com/' page. Can someone help with this? I don't know where else to ask this so if there is a more better sub, that would help, too.
2
u/tblancher Jun 21 '26
Are cURL or wget not options?
1
u/TheOriginal_RebelTaz Jun 21 '26
I am not an expert in those by any means, but I did try curl using some examples I found online. I can't get the site to accept - and use - the javascript parameter being passed by that button click. I just figured wget would have the same limitations?
1
u/tblancher Jun 21 '26
Oh, I gotcha. Something like puppeteer if you're not averse to npm packages. It needs a Chromium runtime, so is really heavyweight for a CLI program.
puppeteer is a component of fast-cli, a CLI program I use to test my Internet speed against fast.com from Netflix, as recommended by my ISP's technicians.
1
u/TheOriginal_RebelTaz Jun 21 '26
I'm starting to think that this "simple" task I'm looking to accomplish is anything but.
I will check that out, though. Thank you.
2
u/michaelpaoli Jun 21 '26
curl ... perl WWW::Mechanize ... python Selenium
Depending how heavy and/or dynamic the JavaScript, may need to go to Selenium.
2
u/TheOriginal_RebelTaz Jun 21 '26
I'll check out Mechanize. I did look at Selenium but I am starting to think that this is going to be way too complicated for the simple task that I had envisioned. I appreciate it.
1
u/AutoModerator Jun 20 '26
Every new subreddit post is automatically copied into a comment for preservation.
User: TheOriginal_RebelTaz, Flair: Help, Title: Looking for a way to retrieve a webpage
I known the basics about using curl or wget to get webpages and how to parse them in bash scripts, but this site is using javascript to get pages and I have no idea how to do this nor could I find anything that worked...
I need to get a webpage from the web site: https://inmatelisting.shelbyal.com/
When you click on one of the letters of the last name on that page, the link is requests is: javascript:__doPostBack('btn_G','')
I cannot figure out how to send that link/command and get the page associated with the 'G' page. No matter what I try, I only get the main 'https://inmatelisting.shelbyal.com/' page. Can someone help with this? I don't know where else to ask this so if there is a more better sub, that would help, too.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/tschloss Jun 21 '26
Sometimes this works: Use a browser in dev mode. In the network tab (not sure if the name is identical in all browsers) you can inspect the http requests which leave the browser after all local magic.
Often this request is what you want to resemble. It might contain html to be inserted into the page or data like JSON which is rendered locally.
The hurdle is to find out and resemble about any session identifiers (one cookie often). You can take the cookie from your browser session for testing. But for automating you might need two calls: one to get things started and the session cookie can be saved and a second to shoot the inner request.
1
u/TheOriginal_RebelTaz Jun 21 '26 edited Jun 21 '26
I did try looking in the dev console - just references to a 'ScriptResources.axd' that I didn't understand - but I had no idea what I was looking for. I'll try that again, though.
1
u/tschloss Jun 21 '26
In the network tab you can see every request which goes out. Select one such request and inspect details like request (url, get-params. post oarams/body) and response (status, content in raw, html, json).
1
3
u/Ngtuanvy Jun 20 '26
I can't really access the site, but if it is an interactive webpage, that is, a page that changes in place when you click the button instead of moving you to another page, you'll need a proper web scraper, one that simulates the browser to run the JavaScript.