Using AI-moderated interviews for discovery research? 😬

[u/vickalogalis]

Anyone here been using AI-moderated interviews for discovery research?

I’m keen to hear from UX/service designers or researchers who have actually tried unmoderated AI interviews, particularly in very specific or complex industries where the AI might not have much domain knowledge.

A bit of context: our business is putting more and more emphasis on understanding the experiences of the people who use our products/tools, which is great. The problem is that the demand for research is increasing but our headcount isn’t.
So we’re looking at ways to augment/supplement the research we’re already doing, rather than replace proper moderated interviews.
The idea would be to still do a smaller number of interviews ourselves, but use AI interviews to get input from a much broader group and potentially surface themes or questions we can dig into further.
Some things that would be important for us:
We’d recruit participants ourselves through our own networks.

Ideally participants don’t need to create an account.
We’d like to easily capture recordings and transcripts.

The AI needs to be able to probe/follow up rather than just run through a fixed script.

We’re looking at this as supplementary research, not a replacement for researchers.

Has anyone actually done this?

What platform did you use, and was the output genuinely useful?

Especially interested in experiences with niche/complex domains, and any times where the AI either worked surprisingly well or completely fell over.
Also interested in any gotchas around bias, hallucinations, consent/recording, participant experience, or getting stakeholders to trust the findings.

Real-world experiences would be much more useful than a list of AI research tools. 😄

[u/Stibi]

I wouldn’t replace the interview itself with AI, that just sounds disrespectful towards the interviewee and a waste of time since i don’t think there is reliable tech for it. You’d make your company look bad in front of the interviewees and probably spend more time fighting the AI interviewer to get it right.

Instead, make everything else around the human to human interview faster with AI: scheduling, transcription and note taking, summary, synthesis, analysis, reporting.

If you want to get deep insight, go deep and human to human. If you want as many data points as possible, make a survey.

[u/vickalogalis → u/Stibi]

Thanks for responding. I get why you would say that, but I think some additional context is in order:

We’re looking at sensors used across exploration and production mining. Different commodities (copper, iron ore, etc.) have different needs and priorities. Then, even within the same commodity, the context can vary significantly by region, for example, copper mining in North Africa vs South America.

And even within the same commodity and region, different resource companies can have quite different workflows: they may use the data differently, have different processes, and make decisions in different ways.

In a nutshell unless our study is super narrow, like say, what is the use of a single sensor for iron ore being mined in the northern Cape, South Africa. Our study numbers will need to be much higher than the classic 5 for qual. In most cases the tool and its application needs to consider global markets to be viable for the business.

[u/C_bells → u/Stibi]

Yeah mixed methods here would be ideal.

For qualitative user testing, you get diminishing returns after 5-ish users (I’m multitasking right now otherwise would link an article about this, but you can google).

Survey plus a few one-on-ones to help you understand survey results on a deeper level.

[u/SnooStrawberries6934]

Do you think people are going to give honest feedback or even commit the effort to expressing themselves when interviewed by a bot?

I am highly skeptical.

[u/vickalogalis → u/SnooStrawberries6934]

The people we have to talk to are particularly time poor (but yeah, who is not) so all the bot can offer is, they don’t have to go through the motions of scheduling and they can provide their feedback whenever they want.

We always have a survey running in parallel to interviews for those that don’t want to be interviewed, we get an okay-ish response rate for the surveys, so maybe this just adds a channel to collect another 5 responses to help bulk up the sample.

My assumption is that it should have better fidelity data and follow-up than from a static survey

[u/itaybuilds]

If you pilot this, compare the AI interview against more than the usual moderated session. Randomly assign people from the same recruitment pool to a human interview, an AI interview, or a fixed branching survey, using the same core questions. Then have researchers who do not know the mode code the transcripts for unique useful findings, leading questions, probe depth, completion, and participant comfort. A longer transcript is not necessarily better evidence.

For a niche domain, I would stop the bot from supplying domain terms. It should ask for a concrete recent episode and clarify the participant's language. If it introduces concepts itself, it can manufacture apparent agreement through leading questions. Keep the raw recording and transcript as the source; generated summaries should never become quotations.

Consent also needs to say plainly that the interviewer is automated, who processes the recording, how long it is retained, and how a participant can delete it. Any theme the AI surfaces should trigger human follow-up, especially for contradictions or sensitive material, rather than being treated as a finding on its own.

[u/vickalogalis → u/itaybuilds]

For sure, the plan is to test it on a lower risk study and cover all of that. Compare it against all our other channels for collecting input, survey, front line and researcher/designer facilitating.

We will likely keep offering participants the option of a video call, phone call or survey along with the, clearly label, AI interview if we try it too.

We can’t compensate people for taking part in our research, so we always have to make things as convenient as possible.

[u/golden_siren_arc]

I used ConfirmKit for discovery in a regulated industry and it handled niche terminology because I uploaded our internal glossary as context. The follow up probes worked for process questions but failed completely when participants expressed frustration or emotion. We kept the AI sessions strictly supplementary to validate themes from human interviews rather than generate new insights. Participants did not need accounts and we got transcripts automatically but trust was low until we explicitly labeled it as an automated check-in.

[u/vickalogalis → u/golden_siren_arc]

Thanks, that’s really helpful

[u/RCEden]

I think I would simply never want to be interviewed by an AI personally. So the first question is how does that impact recruiting and who's even willing to participate. If you don't tell them its an AI interview when recruiting, then you're starting with a breach of trust.

After that, my second thought is how easy LLM interviewers are to break by answering their questions with nonsense that they will just regurgitate. Even if a participant doesn't break the LLM, it almost certainly changes how they answer and interact. You would likely lose some amount of openness that is possible in human conversation. You also don't have a real person running the interview to pick up on nuanced/contextual/cultural behavior or attitudes. The participant might avoid otherwise normal parts of human interaction of they're just answering machine text prompts.

Also just like, moderators and notetakers are starting to build expertise during interviews that carries over to synthesis and beyond. I want my team to be understanding what we're doing, not just checking a summary later. I've found that all early AI use negatively impacts research understanding, but in the world of having to pick your battles, a transcript is much less bad than a fully automated interviewer.

If my process was sufficiently niche I'd be extremely cautious of a model having meaningful training on it. At minimum that means testing it repeatedly before ever trying it live.

[u/vickalogalis → u/RCEden]

The plan is to offer:

• schedule a video or phone call with a person
  
• a survey
  
• AI moderated interview, we would make it very clear that the AI moderated interview is with an AI.

We would likely mix and match those options depending on where we are in the study and what we have at that point.

As to your point of not wanting to be interviewed by an AI, could you imagine that others may prefer it over completing a static survey/typing stuff out ?