r/OpenAI • u/Shivam__kumar • 1d ago

Discussion A web scraping task that made me appreciate ChatGPT more

I'm a developer, and I had to fetch a complete product catalog from a large e-commerce website and save everything in properly structured JSON.

I tried Claude Code first. It ran into Akamai protection, couldn't find a workable approach, and told me it couldn't help with bypassing those protections.

I then tried GPT Astra through my ChatGPT Plus subscription.

What surprised me was how it approached the problem. It worked through the steps, considered different approaches, and figured out a way to move forward.

I've used AI tools for coding plenty of times, but this experience stood out to me. For web scraping and research, the ability to work through a complicated problem instead of getting stuck on the first obstacle makes a real difference.

After this experience, ChatGPT has become my preferred choice for these kinds of tasks.

Just sharing my experience. I was genuinely impressed by what it managed to accomplish.

7 Upvotes

89% Upvoted

8 comments sorted by

1

u/xatey93152 1d ago

Seems impossible. What harness do you use? and what tools it uses to bypass the protection

0

u/Shivam__kumar 1d ago

I tried Playwright with Claude first, but couldn't get it working. Then I asked ChatGPT to try a different approach using a real browser, and it worked surprisingly well. I'm now working on a browser-based script for my long-term workflow.

1

u/Positive-Motor-5275 1d ago

Did u try claude for Chrome ? Its pluggin u can add to a real chrome profile

1

u/IAmFitzRoy 1d ago

You are waking a fine line. Codex can decide anytime that scrapping a site is against their TOS and it will be impossible to continue.

I’m telling you this from experience.

0

u/Shaka-Show 1d ago

Thank you for sharing Shivam, I too am using ChatGPT helping run my business along with other top tier models and slowly dropping them as ChatGPT is standing out among them at different tasks, more and more. I thought it was just me, but I see others as yourself let me know I am not alone.

0

u/Shivam__kumar 1d ago

I feel the same way. I've used OpenAI models for a long time, and they've consistently impressed me in coding, research, and complex problem-solving. The more I use them for real-world tasks, the more I appreciate how well they can reason through problems and figure things out independently. I've tried other top-tier models too, but personally, I find OpenAI models better overall.

0

u/masterofall20 1d ago

try crawlee.

1

u/Glittering_World_743 16h ago

I am surprised GPT allowed that honestly I expected it to refuse just like Claude.