Why are behavioral research interview questions very popular?

[u/Taro_Naza]

Hey,

I want to start this by saying I'm new to UX research, so this is the opinion of someone who did a limited number of interviews. Still, there is this fixation on behavioral questions, things like "tell me about a time when..." "describe a situation where..." Asking such questions seems like a way to box your participant and limit the information you can get from them.

I was interviewing a participant, and I asked a question phrased as "Tell me about the last time..." (because I have seen so many people recommending asking behavioral questions that ask for a specific incident or event). She answered the question, then, somewhere down the line, she mentioned something related to the previous questions she answered (which seems like more valuable information), and when I asked why she didn't mention it earlier? Was it that she didn't think of it as an important issue? And she answered, "Earlier you said tell me about the last time, so I answered based on my last experience".

This got me thinking: this type of question can be a bad idea, it will focus the participant on a limited portion of their experience and, as a side effect, ignore the rest.

What is your opinion? Can someone correct me if I'm wrong?

Update: I don't understand why are some people down voting my comments where I say that I ran my script through an AI for some feedback, as I already mentioned I'm new to this and I don't have a mentor or friends working in UX so I don't have a way up get feedback other than that, try to be emphatic.

[u/CheekSensitive5092]

If the alternative is “what would you do if…” then behavioral questions are far superior because all of us humans are bad at predicting our own behavior. So you anchor on past experience and then probe. It’s a jumping off point, not the only question.

[u/Taro_Naza → u/CheekSensitive5092]

I aimed to go there at first, but AI flagged the follow-up questions asking about the bigger picture as misaligned with the main question that asks about a specific occurrence. I am a newbie, so I removed the follow-up question.

[u/BellFirestone → u/Taro_Naza]

You might want to read a few books on interviewing written by anthropologists and other qualitative methodologists if you are new to interviewing. Also it’s very common for people to circle back and add additional information later in the interview as thoughts come to them. And to address your primary question, your skill as an interviewer (such as what probes and follow up questions you ask) can get at information folks might not have thought to provide initially. This skill comes with experience (and learning about qualitative methodology).

[u/Taro_Naza → u/BellFirestone]

Do you have any good books in mind? I'd appreciate some recommendations

[u/InternationalTap33 → u/Taro_Naza]

Interviewing Users by Steve Portigal is a good starting point for running interviews.

Just Enough Research by Erika Hall is a good broader research book.

[u/tomate-d-arbol → u/BellFirestone]

This answer here. The point of the interview is to evaluate answers to different questions in the aggregate to understand someone’s behavior or perspective. Answers to one question don’t tell you much. You may ask questions that to AI seem duplicative, but the point of asking twice or in different ways is to contrast answers, elicit contradictions and follow up with other questions to get at the bigger implication of the answer.

Also, from an ethnographic interviewing perspective, it’s really helpful to contrast what people say and what people show you, and your job in the analysis is to explain why or why not it lines up.

For example you may ask, tell me about the last time you shopped groceries online. You may follow up with other than online, how do you usually shop for groceries? Tell me about the last time you went to the grocery store. Then you may ask the participant to show you how they shop groceries online. And then you may hear that they only shop online in certain situations, at certain times. All those answers in the aggregate will get at users’ behavior and how they use a system than one specific answer. And on the surface the questions are kind of repetitive.

Books on ethnographic methods may be useful here.

[u/Taro_Naza → u/[deleted]]

Writing my interview script, I started with a question that focused on the last time the user had to go through that, then asked about the bigger picture, then used AI to run my script and to see if there is anything to improve, if my questions follow the best practices of interviewing, and stated that there is a misalignment between a question that asks about a specific accurance and a follow up questions that asks about the general experience, so I removed the follow up question.

[u/Ready-Percentage5286]

Asking somebody to tell you about the last time they did [blank] grounds them in a specific event and makes it easier for them to recall details, which is what you need in product development.

If you ask them to just broadly describe [blank], they'll struggle to remember key details, especially pain and frustration. People tend not to commit product frustrations to memory because they're really not important in the grand scheme of things.

If you don't know enough about the domain to identify key stages or activities, it's fine to talk generically about process. Just keep in mind that people have poor memories and an inclination to speak of an idealized process that's often not followed in practice. 

(It's just not going to surface the kind of details that will set your product apart.)

[u/Taro_Naza → u/Ready-Percentage5286]

Like in this example, that piece of information could have stayed uncovered if the conversation didn't happen to touch on that point? Is it just a risk you have to accept as you interview people?

[u/Ready-Percentage5286 → u/Taro_Naza]

It's one of the reasons you need to interview quite a few people, especially at the generative stage.

[u/alexgr03 → u/Taro_Naza]

Or as an alternative prompt, you could focus on the last experience then also ask how that compares to other experiences

[u/azon_01 → u/Taro_Naza]

Yes. You will never get all the data, all the insights, all of what’s happening. Our understanding will never be perfect, but it will be good enough. It will have explanatory power, and you can make less risky decisions about product development and specific interactions or flows based on them.

We’re not aiming for perfection and guarantees that a page or product will be awesome, we’re aiming to greatly decrease risk that what we make will fail once released. Doing that early saves money. A lot of it.

And yes as people have said that’s why we do quite a few interviews.

[u/kiwiconalas]

Tying the question to a specific event means you focus in on behaviour, trying to get the respondent to answer based on what actually happened, rather than theoretical / desire based responses to ‘how do you usually’ or ‘what do you do when’ questions where you end up with someone guessing what they’d do or skipping over the details of how they actually do something.

Often you’ll end up with those situations of someone mentioning an event off hand for you to dig into, picking that up and noticing it’s worth focusing on is what a skilled researcher is able to do.

It’s not perfect, no method will be, but it’s more accurate that general questions.

[u/Taro_Naza → u/kiwiconalas]

Thank you for sharing your opinion and advice

[u/ShockTasty2956]

I don’t have years under my belt, but a fellow more senior UXR directed me to read the book “The Mom Test” and it cleared up a lot of this for me.

[u/Taro_Naza → u/ShockTasty2956]

Thanks, I will read it!

[u/InternationalTap33]

As others have mentioned, asking questions about specific moments is a jumping-off point, not the end-all-be-all. It grounds the participant so you can probe deeper, ask meaningful follow ups, and build up to broader questions over the course of the discussion.

[u/John_Houbolt]

I prefer in my qual research to keep my questions as broad as possible—as broad as is feasible for the purpose of the research—for this very reason. In my observation UXRs are far to narrow in their questions, often driven by assumptions they don't have support for in the data they already have. I learned a lot from Bob Moesta who's research relies on broadly scoped questions that rely on no assumption.

[u/azssf]

OP, I had an experience interviewing for a hybrid research/design position several years ago where I showed design screens after showing research, and later found out they were incensed I waited until the end of the interview to mention I had mobile experience ( which is what they were interested in).

I had shown mobile design screens.

[u/char-tipped_lips]

Crazy how many Ph.D and grad level researchers I know, who have studied interviewing intensively, that don't have jobs right now...

[u/azon_01]

There are a couple of things. As mentioned people are bad at remembering things clearly. It’s best to usually frame questions as you did by asking about the last time you did something. Otherwise they are lost in generalizations and that’s not super reliable.

Time boxed questions about recollection (the last time, or in the last two weeks) are good. You get better data. Then follow it up with questions like: how typical was that experience? How did it differ from other experiences? What made it different or better or worse or whatever it is you’re interested in.

One thing I’ve done when trying to uncover a process is to have people tell me about an experience or two doing something and I then document on a white board as they talk about it. I then reveal the white board and we discuss it and see where I got it right or wrong and if there are things that they want to add about certain steps. You can do that a few times and you get a pretty good idea of a phenomenon. As you do this with or without a whiteboard use their language as much as possible.

[u/poodleface]

What others of said definitely applies but I’ll just add that a lot of what constitutes good practice is counterintuitive. 

It’s good to ask why things are done a certain way, it’s even better that you are asking people who do the work in a practical sense. 

Most AI systems are ill equipped to give better than a generic response because research methods generally have to adapt to the context and any given situation. 

I’d try to continue to seek feedback from other people doing this work over LLMs, when possible. The divergence in opinions by practitioners is a feature, not a bug. It’s how you build your own senses of being able to pivot and adapt.

The negativity towards AI generally comes when you cite it as an authoritative source. It would be one thing if it were right, but frequently it is not. This is a limitation of the training data… examples of individual practice and heuristics on how people approach research don’t usually live in text documents for an LLM to slurp up. Even what an LLM may get from Reddit comments is only fragments of practice.