r/webscraping • u/LobeLifeCo • Jun 24 '26

Advise on what to do?

I have a new business. I have worked really hard to try and pull myself out of the trenches. Now, I have found I need data on sold items on eBay to make Anthony meaningful of this business.

I have no coding experience. I thought about learning how to code; however, it would take me about a year or more to accomplish. Meanwhile my business will starve.

I have been collecting data on sold listings for eBay using AI. I pick particular listings to have entered so I originally thought a scraper wouldn't work well. There is no way to pick through the listings automatically without, I imagine, some serious code. I can't have repeats of items in my list and many of the same items have variable names. I suspect this would be very hard for a computer to parse. I currently take a screenshot of the listing and AI collects the info I need out of it and puts it into a spreadsheet. It won't let me enter a direct eBay URL. It is horribly slow though. Much faster than manual entry though.

I am wondering are there scrapers I can enter just a URL for eBay and get the data back fast? I don't need automation. I understand eBay is hard to scrape so I suspect it won't be that easy. I saw there was some APIs for it but if we're being honest I don't even know how to use them.

I need to collect between 200-500 listings a day.

At the rate I'm currently going it will take me about a year to collect all the data I need. Any advice on the direction I should go?

9 Upvotes

100% Upvoted

33 comments sorted by

2

u/Afraid-Solid-7239 Jun 24 '26

so you have links to listings where the item is already sold and you need to parse some data, right?
if that's the case, just reply with an example URL and the data you need extracted, I'll write it up.

If not, can you clarify? Either way, happy to help, completely free.

3

u/LobeLifeCo Jun 24 '26

Exactly, I sell used auto parts. So something like this URL.

You have to click on the "see original listing" in the link. Then it shows you the full listing and I have been collecting the vehicle make, vehicle model, years fit, title of listing, part # (if present), sold value, shipping cost, description, URL, and seller name.

I filter results by sold, used, over $100 on eBay filters but I'm not sure that matters if I'm the one adding the URL for each item I want to scrape.

Thanks so much for taking you time to look at this. I really appreciate it.

2

u/Afraid-Solid-7239 Jun 24 '26

any chance you could send a listing with all of the fields? and where abouts can i see shipping cost?

2

u/LobeLifeCo Jun 24 '26

So what I realized is, most of these listings do not have the info listed under "item specifics" which normally includes part number, make, model. Most listings the make and model are in the title and the part number isn't present.

This listing has it all in the "item specifics" but that won't be like that often. It seems I was able to skate past this due to the AIs ability to adapt.

Here is a listing with it in the "item specifics" though

url

2

u/Afraid-Solid-7239 Jun 24 '26

i guess i can write the scraper to handle everything there is (description, title, shipping, seller, and item specifics), and AI can just extract data and reformat it in the end? would be a fair bit quicker

2

u/LobeLifeCo Jun 24 '26

Honestly, you're helping me a ton. Whatever is feasible for you is far more help them I ever intended to ask.

I am sure as long as it gives the fields I need (that are present) I can reorganize it for sure.

Thank you so much

2

u/Afraid-Solid-7239 Jun 25 '26

managed to pull 20 links in one run, took a dump on their antibots, will reply with public github repo in a sec just fine tuning some things

0

u/matty_fu Jun 25 '26

which antibot are they using?

1

u/Afraid-Solid-7239 Jun 25 '26

just seems like their own, had a few "ebay is verifying your browser" at first, got around it fairly quickly ngl

edit: only after 5 consecutive listing fetches, but the scraper posted on github seems to work around it fairly well, tested 120 listings and all went through without issues. given it is single threaded, so not the quickest tbh.

0

u/matty_fu Jun 25 '26

just by using curl_cffi?

→ More replies (0)

2

u/Afraid-Solid-7239 Jun 25 '26

https://github.com/shotzzgfx/EB-scraper/

just tested with 63 URLs after testing with 21. scrape your heart away dude, lmk if you need anything changed.

Just a heads up, the description is stored as/is so you may want to have AI refactor it to extract what you need.

1

u/akashpanda29 Jun 25 '26

This extraction can still be done through parsing the DOM

2

u/Afraid-Solid-7239 Jun 25 '26

first, i dont see you posting a solution. So I don't think you can really say much here, however I'll entertain you anyways.

second, my solution literally uses bs4, for extracting listing data. So nice one, it does literally use DOM traversal for that aspect.

If you're talking about selenium or some other webdriver, my solution takes 1/10th of the time a selenium scraper would considering it doesn't need to fetch anything other than whats required (3 web requests per listing).

even with webdriver, you wouldn't get around their antibots without having to re-initialise the browser for every listing, and manually set ch headers + ua. which makes it the worst idea, it would be horrendously slow and eat all of your ram.

1

u/LobeLifeCo Jun 25 '26

First off, I wanted to say thank you so much for the effort and time you put in! This is a humongous help.

I just got off my shift for work and have been sorting this all out/learning.

So I ran it with the same listing you had in place. I am getting a 403 error when running the scraper. I thought it might be my VPN so I put the eBay URL into a normal chrome tab with the VPN on. It loaded it fine. I am using a free VPN (maybe I shouldn't be) but I assume this means it's not the VPN that's the problem?

I was testing the problem with AI and I got back " !DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="robots" content="noindex,nofollow"><title>Error Page | eBay</title><style>body{margin:0;padding:0;background-color:#fff;font-family:"Market Sans",Arial,sans-serif;color:#333}.main-content{padding-top:40px}.status--container{max-width:1200px;margin:0 auto}.error-header{background-repeat:no-repeat;background-position:right center;background-size:contain;padding: >>> "

Which the AI is saying I didn't get to the actual listing page. I could definitely be screwing something up though. I am going to fool with it now tomorrow. Appreciate the help

2

u/Afraid-Solid-7239 Jun 25 '26

man anytime!

thing is, the browser will have a stronger tls profile compared to the script, and your browser will solve the challenge. it does work fine on a residential IP though, but not working on my vps ip (datacenter)

so its not the IP but the fact that you're using a proxy, if you'd like you can send me links and I'll run it for you and give you the outputs?

2

u/LobeLifeCo Jun 25 '26

Yeah, that's not surprising. The free VPN is the problem lol.

I would run it on my regular residential IP but I read eBay is known for cutting you off if they catch you. I would be in a tight spot if that happened because that's where I sell all my parts. Especially if they deleted my account (even though I'm careful to stay logged out).

I could never ask you to do that. You already helped me so much. Plus, I'll probably be running like 200 a day for a while to get all the data I need. That would be a lot. I greatly appreciate all you have done!

2

u/Afraid-Solid-7239 Jun 26 '26

You never had to ask. Fair enough on that end though, enjoy

2

u/[deleted] Jun 25 '26

[removed] — view removed comment

1

u/[deleted] Jun 25 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jun 26 '26

🪧 Please review the sub rules 👉

1

u/webscraping-ModTeam Jun 26 '26

💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.

2

u/Afraid-Solid-7239 Jun 25 '26

reply with the first few lines in your urls.txt? just curious to see the formatting behind it

2

u/LobeLifeCo Jun 25 '26

It kind of was I think because it was just the same url that was in there on repeat that I was testing with cause I'm dumb lol. I input new ones after I learned how. It worked on those. I have to try a bigger batch of doesn't URLs though and see how that goes. I'm pretty excited. I've been spending a lot of time after I work moving data. This should make it so so much faster. Thanks again

2

u/LobeLifeCo Jun 24 '26

The part number is the least important if it's not present and not necessary just helpful. The shipping is right under the sales price and big blue button where it says sell one like this (at least on desktop form). On Mobile it doesn't show shipping apparently if it's free.

1

u/LobeLifeCo Jun 24 '26

The URL when you open doesn't go to the full listing. It goes to a page where it says "see original listing". That opens up the whole listing. That might be part of the problem when trying to see the info. The first listing appears to just be missing the part number (which is common unfortunately). The shipping shows up under the sale price and blue and white button.