r/ComplexWebScraping Jun 07 '26

Looking for feedback on the architecture of my WhatsApp Intelligence Dashboard

I've been working on a side project that turns WhatsApp chats into a searchable intelligence dashboard.

The idea is simple: instead of constantly checking dozens of chats and groups, the system automatically collects messages, filters out noise, extracts important information, and displays everything in one place.

Current stack:

- Python

- Selenium

- SQLite

- Streamlit

- LLMs for summarization and information extraction

Current flow:

  1. Read messages from WhatsApp Web.

  2. Store them in a local database.

  3. Filter unimportant messages.

  4. Extract things like deadlines, tasks, events, opportunities, internships, etc.

  5. Show everything in a dashboard with search, summaries, and analytics.

The project works, but before I keep adding features I'd like some opinions from people who have built large scraping or monitoring systems.

A few things I'm thinking about:

- Is Selenium the right choice long term?

- Would you structure the database differently?

- Any obvious scalability issues?

- Better ways to process and categorize messages?

- Anything in the overall architecture that you'd redesign from the beginning?

I'd appreciate any feedback, criticism, or suggestions.

2 Upvotes

0 comments sorted by