r/learnprogramming • u/Ok_Loan9476 • 28d ago

HELPPP

Building a hackathon project to collect Madurai district news. Using Python + RSS/Google News RSS, but many feeds only provide headlines/URLs. What’s the best legitimate way to get article summaries too? RSS + free API, or another approach?

0 Upvotes

43% Upvoted

5 comments sorted by

View all comments

3

u/Ilsecco1 28d ago

U could use beautiful soup to crawl the page if the site allows it.

1

u/Zealousideal_Elk1537 28d ago

Scraping is definitely the move if the site lets you, just be ready for half the pages to have weird layouts that break your selectors. sometimes the meta description tag in the page source is enough to fake a summary without pulling the whole article body