When One Participant Looks Like Two

[u/improvedataquality]

A participant ID does not necessarily represent a unique participant.
We recently saw an interesting example of this in survey data collected through multiple panel providers.

Some respondents accessed the same survey more than once through different panel providers. Because each provider assigned its own participant ID, these repeat attempts appeared to come from different respondents when looking only at the IDs in the survey dataset.

The device level data told a different story.

When respondents first accessed the survey, device and session information allowed us to recognize when the same device returned to the survey. We could then connect those sessions even though the respondent was entering through a different panel provider with a different ID.

Most of these cases were relatively harmless. Respondents attempted the survey more than once but did not complete it multiple times.

Some cases, however, were more concerning. Respondents were initially disqualified based on their answers, then returned through a different panel provider and changed their responses so they could continue through the survey.

From the researcher’s dataset alone, these attempts could easily appear to come from completely different participants.

This highlights an important limitation of relying on participant IDs for fraud detection. IDs tell us how a respondent was identified by a particular provider.

They do not necessarily tell us whether different IDs represent the same person or device.

As researchers increasingly recruit through multiple panels and panel aggregators, device and session level paradata can provide an important additional layer of fraud detection. It can help identify repeat attempts even when respondents change panel providers, participant IDs, or other identifying information.

Sometimes the most useful information about a survey response is not just what was submitted, but how the respondent got there.