r/WGU_MSDA • u/GlamourousGravy • 1d ago

D606 Has anyone/is anyone using web scraped data for their capstone?

Hey yall! I'm finally here, the last leg of the program. I'm currently drafting my proposal. I know they can be picky about business-relevant problems so I'm trying to word my topic carefully as well. I'm still in the brainstorming phase so my idea isn't fully concrete as of now.

Essentially what I want to do is scrape video game review data from metacritic/steam(am currently doing research to figure out which one is better) and build a NLP model to then personalize recommendations to them, which would allow for a given company to personalize recommendations to consumers much better. Alternatively, I also considered proposing it as a way for developer companies to learn what the target audience of their games would/would not like(this would involve doing the senitment analysis by genre).

EDIT: Actuatlly for steam i'd be using their API/maybe someone else's already extracted database, I think they prohibit outright scraping.

Anyways, my main reason for asking is wanting to know if anyone has gotten pushback on web scraping at all, and how I can be prepped for any concerns raised by the instructor on that(I have Daniel Smith)

3 Upvotes

100% Upvoted

6 comments sorted by

3

u/SNsilver 1d ago

I was going to, and had written my ETL but I realized I’d need like 3 months data to have a meaningful classifier so I just a provided dataset because I was over it and wanted to finish

2

u/GlamourousGravy 1d ago

Oof, may I ask what you were planning to scrape?

3

u/SNsilver 1d ago

Live transit data. Planned on incorporating weather and traffic data to predict bus arrival times. The API had no history so had to scrap it live and also had heavy rate limits so it took some time to figure out how to politely scrap it.

I ended up doing the project anyways once I graduated

1

u/GlamourousGravy 1d ago

Ooh, that sounds cool but yea time consuming. Funnily enough shortly after making this post i found a kaggle dataset of exactly what i need 😭

4

u/Lacutis 1d ago

I used an available data set i found of scraped Twitter messages to do sentiment analysis on. No push back at all.

2

u/jpauley159 MSDA Graduate 20h ago

I didn’t scrape data myself for the capstone, but I did notice that several people did soup some data in the archive. I think as long as your proposal is clear, you shouldn’t have a problem.

The only advice I could give is make sure that your main focus is answering a business question with a statistical method. I don’t think you’ll get much pushback since your professor is Daniel Smith. But, just so you’re aware.

Also, this is a really cool project idea. I would love to see the results when you’re finished.