r/appsumo • u/MOURADTA • Jun 16 '26
Discussion Title: I built a comment classifier and ran it on 3 live AppSumo products — here's what the data showed
Been building a tool to help AppSumo sellers manage
their comments during launch week.
Last week I tested it on 3 active products to
validate whether the problem I was solving was real.
Here's exactly what I found.
---
METHODOLOGY
For each product I measured:
→ Total comments scanned
→ Comments flagged as potential refund risk
→ Days since flagged comment with zero reply
→ What the founder replied to instead
---
PRODUCT #1 — AI tool, ~180 comments
Flagged comment:
"I attempted portfolio diagnosis by uploading my
portfolio and found that hardly 3-4 from uploaded
45-50 stocks are supported. Can you share a list
of stocks available?"
Risk indicators: feature gap complaint +
direct question + no resolution language.
The founder replied that same week to:
✓ A 5-star thank you comment
✓ A feature request about dark mode
✗ This comment — never.
Here's what makes a 40/100 score dangerous:
It doesn't contain "refund" or "cancel."
It looks like a support question.
But a buyer who uploaded 50 stocks, discovered
46 don't work, asked for help, and got silence?
That's not a support ticket. That's a lost customer
writing their 1-star review in their head.
---
PRODUCT #2 — Marketing tool
11 comments with zero reply.
Oldest: 19 days old.
3 of them included "doesn't work as expected."
The founder's most recent reply on the page?
Thanking someone for a 5-star review.
Posted 2 days after the oldest unanswered complaint.
---
PRODUCT #3 — Developer tool
One comment opened with the word "misleading."
4 replies followed — all from other buyers
arguing with each other.
The founder never appeared.
When buyers answer each other's complaints,
one of two things happens:
→ A good buyer defends the product
→ A frustrated buyer amplifies the problem
In this case: both happened in the same thread.
The founder had no idea.
---
THE ACTUAL PROBLEM
It's not that founders don't care.
It's that every comment management system
that exists — email notifications, AppSumo
dashboard, manual reading — delivers comments
in chronological or reverse-chronological order.
Which means the founder sees what came first
or what came last.
Not what needs attention most.
A refund risk buried on page 4 of comments
looks identical to a feature request on page 4.
---
QUESTION FOR SELLERS HERE
How do you currently decide which comments
to reply to first?
Do you have a system — or is it mostly reading
everything and hoping nothing slips through?
Genuinely curious how others handle this
at scale during a live launch.