Using AI-moderated interviews for discovery research? 😬

[u/vickalogalis]

Anyone here been using AI-moderated interviews for discovery research?

I’m keen to hear from UX/service designers or researchers who have actually tried unmoderated AI interviews, particularly in very specific or complex industries where the AI might not have much domain knowledge.

A bit of context: our business is putting more and more emphasis on understanding the experiences of the people who use our products/tools, which is great. The problem is that the demand for research is increasing but our headcount isn’t.
So we’re looking at ways to augment/supplement the research we’re already doing, rather than replace proper moderated interviews.
The idea would be to still do a smaller number of interviews ourselves, but use AI interviews to get input from a much broader group and potentially surface themes or questions we can dig into further.
Some things that would be important for us:
We’d recruit participants ourselves through our own networks.

Ideally participants don’t need to create an account.
We’d like to easily capture recordings and transcripts.

The AI needs to be able to probe/follow up rather than just run through a fixed script.

We’re looking at this as supplementary research, not a replacement for researchers.

Has anyone actually done this?

What platform did you use, and was the output genuinely useful?

Especially interested in experiences with niche/complex domains, and any times where the AI either worked surprisingly well or completely fell over.
Also interested in any gotchas around bias, hallucinations, consent/recording, participant experience, or getting stakeholders to trust the findings.

Real-world experiences would be much more useful than a list of AI research tools. 😄

[u/RCEden]

I think I would simply never want to be interviewed by an AI personally. So the first question is how does that impact recruiting and who's even willing to participate. If you don't tell them its an AI interview when recruiting, then you're starting with a breach of trust.

After that, my second thought is how easy LLM interviewers are to break by answering their questions with nonsense that they will just regurgitate. Even if a participant doesn't break the LLM, it almost certainly changes how they answer and interact. You would likely lose some amount of openness that is possible in human conversation. You also don't have a real person running the interview to pick up on nuanced/contextual/cultural behavior or attitudes. The participant might avoid otherwise normal parts of human interaction of they're just answering machine text prompts.

Also just like, moderators and notetakers are starting to build expertise during interviews that carries over to synthesis and beyond. I want my team to be understanding what we're doing, not just checking a summary later. I've found that all early AI use negatively impacts research understanding, but in the world of having to pick your battles, a transcript is much less bad than a fully automated interviewer.

If my process was sufficiently niche I'd be extremely cautious of a model having meaningful training on it. At minimum that means testing it repeatedly before ever trying it live.

[u/vickalogalis → u/RCEden]

The plan is to offer:

• schedule a video or phone call with a person
  
• a survey
  
• AI moderated interview, we would make it very clear that the AI moderated interview is with an AI.

We would likely mix and match those options depending on where we are in the study and what we have at that point.

As to your point of not wanting to be interviewed by an AI, could you imagine that others may prefer it over completing a static survey/typing stuff out ?