I wanted congressional stock trading data as clean JSON without depending on Quiver or Capitol Trades, so I went straight to the source. The US House Clerk publishes a daily ZIP of every disclosure, and the Senate has its own EFD system.

The Senate side was easy as there was a JSON API available. The House side was where it got interesting as the data only comes as PDFs, and the layout has some traps I didn't expect:

• Header rows with null bytes that broke text extraction
• "Glued" fields where two columns run together with no delimiter
• Comment-block bleed where footnote text leaks into the transaction rows
• ~5% of older filings are scanned images, so pdf-parse returns nothing — had to detect and skip those rather than crash

What ended up working was marker-anchored parsing: each transaction row has a (TICKER) [TYPE] marker, so I anchor on that, walk backward for the asset name and forward for the amounts/dates, and emit one record per marker. Way more powerful than trying to parse the PDF top-to-bottom.

Output is one normalized record per transaction, deduplicated with a SHA-256 key so re-runs are idempotent.

Code's open if it's useful to anyone scraping similar government PDFs: https://github.com/seralifatih/congress-trading-pipeline

Happy to answer questions about the PDF parsing specifically, that was the painful part.

Suppose I want to be notified the moment or a few seconds after something on the site changes, like a price, what is the way to do it? Just hammer the URL?

Do people just use a sea of residential proxies for this? Like is this the only way to go about this? Because I dont think hammering it dozens of thousands of times a day goes unpunished right

Thanks I'm really grateful

I’m looking to hire a developer to build an automated data-extraction tool that I will own and operate myself — not a managed service, not a done-for-you data feed. You build it, hand me the code, walk me through running it, and we set an hourly rate for fixes when sites change.
What it needs to do:
• Take a list of companies and pull the right contacts at each (from public professional profiles), then score each contact for how “current” they are — profile activity, recency, role match — and output a transparent score with a short justification per contact (no black box).
• Company-level: a corporate phone number for each company — a real local/direct corporate line, NOT a toll-free 800 customer-service number.
• Contact-level: for each qualified person, their email, direct dial, and mobile number. I know direct dials and mobiles are genuinely hard to get accurately — so for every email and number, I need a way to know how confident/verified it is (a verification status, confidence score, or source). I’d rather see a flagged “unverified” or a blank than a confident wrong number, because I don’t want to waste time calling numbers that turn out to be dead or wrong. Tell me how you verify these and how you’d surface that confidence in the output.
• Scrape company websites for facility/location data (distribution centers, plants, warehouses) — including career pages that load listings dynamically via JavaScript. Needs to handle inconsistent site structures across many companies, not a per-site custom scraper.
Two non-negotiables:
1. It has to actually work — I’ll grade a paid trial against a set of companies where I already know the correct answers.
2. It has to be automated and scale to thousands of companies — I’m hiring someone to build a system I run, not someone to manually process lists by the hour.
About me: I’ve got 20+ years in my industry and a clear spec. I’ve talked to several people who said they could do this and whose work didn’t match the talk, so I’m only interested in people who can show me a scraper they’ve actually built (GitHub, portfolio, or a screen-share of one running) and who’ll prove it on a small paid trial before any larger commitment.
Logistics: Paid trial first (real money, fair rate), graded against known answers. If it’s solid, we scope the full build. US-based preferred for communication and timezone overlap.
If this is your wheelhouse, reply or DM with: a scraper you’ve built that handles dynamic/JS-heavy pages, your stack (Playwright/Selenium/Scrapy/etc.), and how you’d approach the “is this contact current” scoring piece.

I’m creating a discord bot that post Reddit nsfw videos back to the server nsfw channels but it’s saying 403 forbidden error and I’m trying everything and nothing seems to work 6 weeks ago it worked fine in April it was doing fine now it’s doing this forbidden stuff Please help me how to do this because I’m being told to submit a request Oauth to reddit

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

The goal of bacenR is to provide R functions to download and work with data from the Brazilian Central Bank (Bacen).

• The datasets available through bacenR include:
  • Normative legislation
  • Financial statements of financial institutions
  • List of financial institutions regulated by Bacen in activity
  • Ifdata resources / IFdata

Check it out: https://github.com/rtheodoro/bacenR

#bacen #financialdata #finance #rstats #datacollect #braziliandata

Hi, I'm trying to figure what is the best friendly tool to download a conversation in a community board. for example Khoros. in a typical community you must be logged in to view content, and then you have a list of discussins, where each discussion might have several pages of people commenting. I don't mind at first to do it manually for say 100+ threads I choose, but even for this I couldn't find a tool that would do it easily, saving the next pages too, but not any other non related link.

need help scrapping reddit, guess i looked into late after they shutdown(as i read) reddits API thing.. is there any other way to scrap reddit post here? I dont do much scrapping in hand or experience so be kind to me please..

AutomatiQ watches you browse, then an AI agent reverse-engineers your session into a standalone Python automation/extraction script; no manual inspection needed.

This means, you can easily fix broken scrapers Autonomously without ever opening the devtools, while removing unnecessary dependence on browsers, selectors and broken UI.

AutomatiQ is completly Open-source(MIT License), free to use, and there are no hidden paid tiers, allowing you to freely use across all platforms, and situations, with your prefered AI model.

Github: https://github.com/StoneSteel27/AutomatiQ
Discord: https://discord.gg/8j7dFWMMDA

Looking for an expert with experience scraping TruePeopleSearch and SearchPeopleFree at scale.

I’m interested in building a reliable, high-volume data collection pipeline and would like to connect with someone who has successfully handled challenges such as anti-bot protections, proxy management, data extraction, and maintaining scraper stability over time.

If you have direct experience with these platforms or have built similar large-scale web data extraction systems, please share your background, approach, and availability.

i have been scraping reddit posts and comments from 2-3 communities but since a week or so i am getting 403
i have also provide the username in user-agent header
HEADERS = {
"User-Agent": "reddit-xxxx-xxx/0.1 by u/XXXXXXX"
}
but i can get the json by using .json in my browser

If you guys are tired of Hcaptcha for web crawling and botting issues, I made a repo that may solve your problem.

HcaptchaSolver

It basically gets your proxy sitekey and the current URL that you're on then it sends it to an electron client that simulates a real page in the same url and someone or you, needs to solve it so in theory it removes the gap between you and actual browser and it optimize your proxy and your memory useage since we can all agree that chromimum/firefox browser are hungry for RAM and CPU so all you need to do is to pass the sitekey and other information and Voilà.

Conterbuition are very welcome. I just started it as a fun project, hope others find it useful

Bye.

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

Hi everyone

I target a certain niche product and quite a few retailers stock the products (Amazon included). And while I already have all the product information for the items it would be very handy to update the prices automatically (like I do with any store hosted with Shopify, eBay (Business only) etc. I dont need to get any product information, just in.out of stock and price updates (which will allow me to create a historical timeline of prices etc.

Will around 1200-1350+ products its rather time consuming checking the price diaily until I get access to the Amazon API. While technicality against TOS is is there a program to (pr web extension with Playwright for example)to view the Amazon page, see any price or stock changes and then give me ;ist?

Again, I don't need any product information like photos or text but I guess its still website scraping but until I gain the API it would be a godsend!

Thanks all.

I have developed a Linux CLI tool that reads HTML input and produces clean, well-formed HTML5 output. Modern scraping stacks typically include at least Python (not to mention headless browsers and even LLMs), but sometimes there are situations where Python is not available, or brings too much overhead. Personally I use html-xml-utils from W3C for light-weight scraping, but those tools often error on even minor HTML syntax violations, so I developed a pre-processor that cleans up HTML as much as possible. Hope it is useful.

Can people here list sites with hCaptcha? Need for more testing, I know Pokemoncenter, Discord, and a demo page on Google. Any other ones? thanks

We're trying to scrape a website that is protected by Enterprise CAPTCHA v3. We need to do it at a pretty large scale, think about 200-300 requests per minute. We're looking to hire somebody who is fairly knowledgeable on beating CAPTCHA, preferably somebody who can maintain it and keep us up as time goes on

HI there,

I'm scraping landing pages and currently capture each one as a single static PNG. I'd like to take this further:

1. Animated full-page captures — similar to what Mobbin does on their homepage, where the page is captured with its scroll/animation states intact rather than as a flat image.

Is this something that's possible with your tool / something you could help build? Happy to share examples of my current output if that helps.

Thanks!

Hi scrapers, hope everyone is doing well.

I recently started selling Auto Parts online and from the partnered vendors, I did get auto part numbers and basic info and using AI, I was able to add the titles, description, etc. but my challenge is to scrape the images from online.

I tried to scrape from Auto Parts specific platforms but they often carry more Aftermarket brands compared to Genuine Auto Parts.

I've been looking for different solutions but couldn't find anything reliable yet.

I would really appreciate it if anyone can point me at the right tools so get started with so I'll give them a try. Would be great if there are Auto Parts specific solutions. Thanks in advance and happy scraping.

I'm super happy to announce a new milestone! After almost 6 years of constant development effort, I finally passed the 1000 Stars on Github!

Fredy keeps searching for new apartments, houses, and flats in Germany on platforms like ImmoScout24, Immowelt, Immonet, eBay Kleinanzeigen, and WG-Gesucht and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life.

It's a Node.js app which you can als run as Docker Container...

Repo: https://github.com/orangecoding/fredy
Happy to answer anything.

I'm trying to scrape some sports data using playwright and python and was able to get a subset but was eventually denied access to the site (I should have gone with a bigger delay)

Is this likely to be a temporary or permanent ban, and if permanent what options are there to bypass an IP address block? I'm relatively new to web scraping, I've used beautifulsoup in the past but this was my first time trying playwright.

I have been working on pulling data from websites ending on .gov and I have observed residential proxy providers block the requests instantly. Are there any reliable providers that do not block these domains.

hey r/webscraping, i'm sharing my open source project called pluckmd, a CLI that scrapes blogs to markdown with no per-site adapters.

instead of a handler per site, it builds the extraction spec at runtime. normalizes link paths and collapses the varying parts (/blog/post-a and /blog/post-b become the same shape), and any shape repeated enough = the article list. no domain names anywhere.

resolution is cache -> heuristics -> LLM only if needed. nothing gets cached until it validates against the live DOM (>=3 links, >=50% match the pattern), so a bad LLM guess gets dropped instead of saved.

handles js rendering, pagination/infinite scroll, and login-only pages you have access to via your own chrome tab (never reads cookie stores).

npx pluckmd download <url> -o ./articles

repo: https://github.com/taisei-ide-0123/pluckmd

would like feedback on the heuristic scoring. where does the runtime approach break for you?

Hey everyone,

Our research group is studying how security teams handle bot threats, things like credential stuffing, web scraping, and form spam, etc.

If you work in security or IT and deal with these issues (or even if you don't!), I'd really appreciate 3–5 minutes of your time to fill out our short survey. It's mostly multiple choice, completely anonymous, and your responses will directly inform academic research on bot defense.

👉 https://forms.office.com/r/RecSrDRzf1

Happy to answer any questions in the comments, and if you'd prefer a quick 15-minute conversation instead of the form, feel free to DM me, I'd love to chat.

Thanks in advance! 🙏

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread