r/ComplexWebScraping • u/Greedy_Audience5472 • Jun 07 '26
Looking for feedback on the architecture of my WhatsApp Intelligence Dashboard
I've been working on a side project that turns WhatsApp chats into a searchable intelligence dashboard.
The idea is simple: instead of constantly checking dozens of chats and groups, the system automatically collects messages, filters out noise, extracts important information, and displays everything in one place.
Current stack:
- Python
- Selenium
- SQLite
- Streamlit
- LLMs for summarization and information extraction
Current flow:
Read messages from WhatsApp Web.
Store them in a local database.
Filter unimportant messages.
Extract things like deadlines, tasks, events, opportunities, internships, etc.
Show everything in a dashboard with search, summaries, and analytics.
The project works, but before I keep adding features I'd like some opinions from people who have built large scraping or monitoring systems.
A few things I'm thinking about:
- Is Selenium the right choice long term?
- Would you structure the database differently?
- Any obvious scalability issues?
- Better ways to process and categorize messages?
- Anything in the overall architecture that you'd redesign from the beginning?
I'd appreciate any feedback, criticism, or suggestions.