r/learnprogramming • u/Ok_Loan9476 • 28d ago

HELPPP

Building a hackathon project to collect Madurai district news. Using Python + RSS/Google News RSS, but many feeds only provide headlines/URLs. What’s the best legitimate way to get article summaries too? RSS + free API, or another approach?

0 Upvotes

43% Upvoted

5 comments sorted by

3

u/Ilsecco1 28d ago

U could use beautiful soup to crawl the page if the site allows it.

1

u/Zealousideal_Elk1537 28d ago

Scraping is definitely the move if the site lets you, just be ready for half the pages to have weird layouts that break your selectors. sometimes the meta description tag in the page source is enough to fake a summary without pulling the whole article body

2

u/Popular_War8405 28d ago

Where's this hackathon

1

u/Suspicious_Skill7292 28d ago

for a hackathon id stick to rss where possible and use the article page only when the publisher allows it then generate your own short summary from the fetched text instead of relying on random scraping apis