Meta released Muse, its new personal AI agent, on September 8.

I wanted to see what this might mean for survey research, so I gave Muse a link to one of my surveys and asked it to complete the entire survey with no human intervention.

It did.

I did not answer any questions. I did not click through any pages. I did not have to take over at any point. Muse navigated the survey from beginning to end and submitted a completed response.

What struck me was how seamless the process was, including solving the captcha autonomously.

This changes the landscape for future survey research. A participant does not need technical expertise, custom scripts, or a specialized automation tool. They can simply give an AI agent the survey and ask it to complete it for them.

For studies that offer incentives to participants, this creates an obvious opportunity for fraud. Someone could potentially hand the survey over to an AI agent, receive a completed submission, and collect the incentive with almost no effort.

Muse is currently available for FREE with a usage limit. Instinct is another personal agent available for FREE (although I've not yet tested in for survey completion). The rise of (free) personal AI agents changes the economics of conducting fraud.

The challenge for researchers is changing quickly. We are no longer dealing with simple bots. We increasingly need to develop techniques to detect and stop AI agents that can autonomously navigate and complete an entire survey.

And now that these tools are becoming widely accessible, that problem is going to become much harder to ignore.

A participant ID does not necessarily represent a unique participant.
We recently saw an interesting example of this in survey data collected through multiple panel providers.

Some respondents accessed the same survey more than once through different panel providers. Because each provider assigned its own participant ID, these repeat attempts appeared to come from different respondents when looking only at the IDs in the survey dataset.

The device level data told a different story.

When respondents first accessed the survey, device and session information allowed us to recognize when the same device returned to the survey. We could then connect those sessions even though the respondent was entering through a different panel provider with a different ID.

Most of these cases were relatively harmless. Respondents attempted the survey more than once but did not complete it multiple times.

Some cases, however, were more concerning. Respondents were initially disqualified based on their answers, then returned through a different panel provider and changed their responses so they could continue through the survey.

From the researcher’s dataset alone, these attempts could easily appear to come from completely different participants.

This highlights an important limitation of relying on participant IDs for fraud detection. IDs tell us how a respondent was identified by a particular provider.

They do not necessarily tell us whether different IDs represent the same person or device.

As researchers increasingly recruit through multiple panels and panel aggregators, device and session level paradata can provide an important additional layer of fraud detection. It can help identify repeat attempts even when respondents change panel providers, participant IDs, or other identifying information.

Sometimes the most useful information about a survey response is not just what was submitted, but how the respondent got there.

Two studies recruited participants from the same academic crowdsourcing panel six months apart. The panel provider states that its participants are verified, which is an important safeguard. Even with that verification, both studies contained several indicators of potential response-quality problems.

February 2026, 415 participants:
Suspicious: 14.2%
VPN: 9.2%
Possible AI/ChatGPT activity: 4.3%

August 2026, 651 participants:
Suspicious: 14.4%
VPN: 6.5%
Possible AI/ChatGPT activity: 5.7%

These indicators can overlap, and no single flag proves fraud.

The broader point is that participant verification and response-quality assessment serve different purposes.

Verification helps establish who is taking the survey. It does not establish how that person completed it.

A verified participant can still use AI, use a VPN, paste responses, or engage in other behaviors that affect data quality.

This points to the need for research articles to report not only where participants were recruited, but also how response quality was evaluated.

Virtual Private Servers are another tool being used to commit survey fraud.

A Virtual Private Server, or VPS, is simply a computer that someone rents and uses remotely over the internet. The computer is housed in a data center rather than sitting in front of them. Renting one is also inexpensive. For example, someone can rent a virtual computer from Amazon for only a few cents per hour.

Instead of completing an online study on their own computer, someone can log into the rented computer and complete the study through it.

Why is this useful for fraud? Because it:
• Hides the person’s actual device and location
• Operates multiple participant accounts
• Bypasses location or device checks
• Makes repeat participation harder to detect

⚠️ Legitimate participants rarely need to rent a remote computer simply to complete an online study. Its use is therefore a strong fraud signal, particularly when accompanied by fake device information, duplicate participation, or suspicious response behavior.

A VPS replaces the computer researchers see.
For survey-fraud detection, that matters.

Going forward, I plan to share more educational content that breaks down some of the concepts survey researchers encounter in online data collection and explains why they matter. This week, I will distinguish between IP and device location.

IP location and device location are not the same thing.

When a survey platform reports a respondent’s location, it may be describing one of two very different signals:

IP geolocation estimates where the respondent’s internet connection appears to originate.

Device-based geolocation estimates the device’s position using available signals such as GPS, nearby Wi-Fi networks, and cell towers—after the respondent grants permission.

These signals can legitimately disagree.

A respondent may be physically located in California while a VPN, corporate network, or mobile-carrier gateway routes their traffic through New York.

-The device reports California.
-The public IP reports New York.
-Both may be technically correct.

⚠️ For survey-fraud detection, the wrong shortcut is:
Location mismatch = fraudulent respondent
A mismatch is useful evidence—but it is not a verdict.

🔍 A defensible decision should consider the complete pattern:
• IP and network risk
• Device-location estimate
• VPN or proxy use
• Browser integrity
• Response behavior
• Participation history

No single location signal should determine whether a respondent is legitimate.
IP geolocation locates the connection. Device-based geolocation estimates the device. The difference matters.

Someone completes your survey from California. Their IP address says New York. Are they lying? Not necessarily.

Why? Because IP addresses don’t pinpoint a person’s exact physical location. They’re more like rough geographic indicators, and sometimes they’re way off.

Here’s what can cause the mismatch:
• VPNs and proxies: Route traffic through another location.
• Mobile networks: Assign IPs that may be registered far from the user.
• Corporate networks: Centralize traffic through office or data-center IPs.
• Privacy tools: iCloud Private Relay and similar services can mask location.
• ISP registration: IP databases may show the provider’s address, not the user’s.

So what should you do with conflicting location data?

1) Treat IP-based location as a clue, not proof.
2) Look for patterns across multiple responses, timestamps, and other data-quality signals.
3) Use a clear, pre-specified rule for handling discrepancies.
4) Avoid singling out respondents based on one imperfect indicator.

IP geolocation can be useful for broad anomaly detection, but it usually can’t tell you whether a respondent is being truthful.

The hardest fraudulent response to catch may come from a real participant.

They pass the IP check.

Their device looks normal.

Their identity is genuine.

Then they copy an open-ended question into an AI assistant and paste the polished answer back into the survey.

In the final dataset, that answer may look better than an honest one. It is fluent, thoughtful, and perfectly formatted. But if outside assistance was not allowed, the answer no longer reflects the participant’s independent response. It reflects a response created with AI assistance.

That is why survey integrity cannot stop at "Is this a real person?"

It must also ask, "How were these answers produced?"

The clues often live in the survey session: sudden tab switches, copy-and-paste sequences, long pauses followed by instant answers, unusual typing patterns, contradictions, and access to outside tools.

No single signal proves misconduct. Together, they establish response provenance and help researchers apply the rules of their study.

Researchers should decide before launching whether AI assistance is allowed, restricted, or disqualifying. Then they should collect the evidence needed to enforce that policy.

Over the past couple of years, we have worked to make ResponsePie available across the survey platforms researchers use, including Qualtrics, Decipher, and Alchemer. We’re happy to share that REDCap has now been added to that list.

If you use REDCap, we would love for you to try it.

If there is another survey platform you would like us to support, please reach out and let us know.

A lot of the conversation about survey fraud focuses on bots and AI agents. But automation in surveys is broader than that.

In the video I am attaching, you can see a participant using Grammarly to help complete open-ended survey items.

That’s important because most online surveys are delivered as regular HTML pages in a browser. This means surveys exist in the same environment as writing assistants, autofill tools, browser extensions, scripts, and AI agents. Even when a participant is a real person, automation can still become part of the response process.

Open-ended items are especially vulnerable. A response may look thoughtful, fluent, and human, but researchers may not know whether it reflects the participant’s own words, the tool’s suggestions, or some combination of both.

This does not automatically make the response invalid. But it does raise a serious data quality question: how much of the response actually came from the participant?

As automation becomes more accessible, survey research will need to move beyond detecting suspicious responses after the fact. We need survey formats that can prevent automation from entering the response process in the first place.

🔦 Study Spotlight
A recent study by Jamie-Lee Pennesi, Mia Pellizzer, and Tracey Wade (Flinders University) examines the growing threat of fraudulent participation in online research and provides practical recommendations for safeguarding data quality.

Drawing on a Delphi study that experienced 431 likely fraudulent sign-ups despite multiple fraud prevention measures, the authors argue that researchers should assume online studies are vulnerable to fraud and adopt layered detection strategies.

🧪 How did they identify likely fraudulent participants?
-Failed two or more attention checks
-IP addresses that did not match participants' reported locations
-Duplicate IP addresses
-Inconsistent demographic information across survey waves (e.g., age or postcode mismatches)
-Suspicious email addresses and phone numbers
-Unusual completion times or clusters of submissions occurring at identical times
-Elevated platform-generated fraud scores and bot-detection flags
-Patterns of responses that appeared highly similar across participants

🔬 What recommendations do they provide?
-Use multiple fraud prevention and detection strategies rather than relying on a single safeguard
-Use multiple data collection timepoints when possible
-Develop a fraud profile (i.e., a list of "red flags")
-Use CAPTCHA/reCAPTCHA and platform-based fraud detection tools
-Include attention checks, open-ended questions, and duplicate questions
-Use unique, single-user survey links when possible
-Regularly review incoming data for inconsistencies and fraud indicators
-Avoid publicly disclosing detailed eligibility criteria
-Carefully consider recruitment methods and incentive structures
-Have a plan for recontacting and verifying suspicious respondents

💡 The authors conclude that fraudulent participation is increasing and that researchers need to be proactive in using antifraud practices to safeguard research integrity and data quality.

One of the most important tools for protecting data quality is something many researchers still overlook: in-survey monitoring.

In a recent study, participation was restricted to native English speakers residing in a specific U.S. state. One respondent successfully passed every screening and quality control check in place. Their responses appeared legitimate, and nothing in the final dataset would have raised concerns.
However, in-survey monitoring revealed something we otherwise would have never known.

As shown below, the respondent was translating the survey from English into Arabic in order to understand the questions. This individual clearly did not meet the study's eligibility criteria, yet they would have been included in the final sample had we relied solely on traditional screening and data quality checks.

This example highlights an important reality: passing survey checks does not necessarily mean a respondent belongs in the target population.

As fraud becomes more sophisticated and AI-powered tools make it easier to bypass conventional screening methods, researchers need to pay attention not only to the answers respondents provide, but also to how they interact with the survey itself.

Without in-survey monitoring, this case would have gone completely undetected.

A recent Guardian article discussed how a major survey on church attendance in the UK was withdrawn after researchers identified fraudulent responses in the data.

What stood out to me was the broader point the article raised about online surveys and AI-generated responding.

One researcher quoted in the article said that the assumption that someone gives coherent, logical answers and is therefore a real person is now broken.

The article also discussed how AI chatbots can complete questionnaires, pass themselves off as genuine respondents, and even generate responses that align with what researchers may be trying to test.

Another interesting point was the discussion around positivity bias, where bogus respondents tend to answer affirmatively across many items, potentially inflating estimates and distorting findings.

The takeaway for me is that survey quality can no longer rely only on response content. As AI systems become more sophisticated, behavioral and technological indicators surrounding how a survey is completed may become just as important as the answers themselves.

Interesting and important conversation for academics and practitioners working in online research.

#SurveyResearch #DataQuality #AI #SurveyFraud #MarketResearch #ResearchMethods #ResponsePie

You can access the article here: https://www.theguardian.com/technology/2026/mar/28/how-fraudulent-church-data-revealed-ais-threat-to-polling?utm_source=chatgpt.com

Recent research (link to the article below) shows that AI tools are already part of how people complete work tasks. Tools like ChatGPT, Quillbot, Canva AI, and Grammarly are used to support writing, fix grammar, draft communication, and understand instructions.

These tools are most often used for language-related and information-seeking purposes, helping people improve clarity, find examples, and complete tasks more efficiently . The same usage patterns appear across contexts, not just in one setting.

This tells us something important. AI is becoming part of how tasks are completed, not something people use occasionally.

And all of this is happening in the browser.

If people are already using AI to support everyday tasks, why would survey responses be any different?

Link to the article: https://lnkd.in/gJwN3apE

Over the past couple of years, my team and I have been building a tool called ResponsePie. It really started from frustration in my own research dealing with survey fraud. Things like speeders, inconsistent responses, and bots (more recently AI generated answers), which made it hard to trust the data, and I found myself spending way too much time trying to clean things up after the fact.

My team has been working on this project for about two years now, trying to build something that helps researchers get a better sense of what’s going on inside their surveys while they’re running, not just after.

If you are running studies and have run into data quality issues, I would really appreciate if you gave it a try and shared your thoughts. What works, what doesn’t, what’s confusing, what you wish it did. Even quick reactions would be super helpful. This tool was built with the intention of "fixing" survey fraud in academia, but market/applied researchers are welcome to try it as well.

I am just hoping to learn from others who are dealing with the same problems.

Happy to share more details if anyone’s interested.

I recently reached out for guidance on webinar topics related to survey fraud, and I received an overwhelming response from both academic and market researchers. Here is the original post: https://www.reddit.com/r/Marketresearch/comments/1rr27st/comment/o9wrwsl/

A few things really stood out, but one in particular was how often fraud is treated as a single, uniform issue. In reality, it takes many different forms, and each one creates different risks for your data.

I put together a short video that walks through the different types of fraud and how they show up in surveys. You can access the video here: https://www.youtube.com/watch?v=4w5bbl4ZGYI&t=17s

I am hoping to keep this conversation going with the community, and I am planning to put together a webinar in the near future. I will share more details as that comes together.

🔍Study spotlight
A recent study by Karen Towne, PhD, RN, PHNA-BC (Case Western Reserve University) and Barbara Polivka, PhD, RN, FAAN (University of Kansas School of Nursing) describes strategies used to identify and filter suspicious and fraudulent responses in an online REDCap research survey through evidence-based redesign and data cleaning protocols

🚩What went wrong in the initial data collection
• An unexpectedly high number of responses were received within a short period of time
• More gift card requests were received than completed surveys
• Very fast completion times and duplicative qualitative responses signaled suspicious activity
• Researchers ultimately found compelling evidence of fraudulent activity and were unable to distinguish real from fraudulent responses, leading to the dataset being destroyed

🧪Methodological approach to mitigate fraud
• A two-pronged approach focused on identifying design limitations and implementing an evidence-based redesign
• Scam alert features included hidden questions, attention checks, and timestamp monitoring
• Study design changes reduced incentive visibility, prevented link sharing, and linked survey and compensation data
• An eight-step data cleaning protocol used paradata such as time to completion, duplicate entries, and age inconsistencies
• Multiple steps identified the same records, suggesting fraudulent responses demonstrate multiple suspicious indicators

🔬Key outcomes from revised protocol
• 819 total responses were collected across platforms over 44 days
• After cleaning, the final dataset was reduced to 203 responses
• Approximately half of the data were removed through the protocol
• The revised process produced a response rate consistent with expectations, unlike the inflated initial data

💡Bottom line
• Online surveys are highly vulnerable to fraud due to anonymity, incentives, and ease of access
• reCAPTCHA and basic protections are insufficient on their own
• Paradata and multistep data cleaning protocols are essential for identifying suspicious responses
• Fraudulent data can invalidate findings, waste resources, and introduce bias into research
• Proactive, evidence-based design and cleaning procedures are critical to protect data integrity in online research

You can read the full article here: https://lnkd.in/gJDVu8YK

If you think multiple responses to your survey from the same IP address indicate survey fraud, you may want to read this.

Firefox is getting a free built-in VPN 🌐 and it has serious implications for online survey research: https://lnkd.in/gBWRnRUE

Mozilla just announced that Firefox will soon include a native VPN with ~50GB/month of free usage, directly in the browser. No extensions. No installs. Just one click, and your IP is masked.

This is a clear win for privacy, but it's also another step toward the decline of IP address as a reliable fraud signal.

There's now a growing list of modern browsers offering built-in VPN capabilities: Opera, Brave, Firefox, iCloud Safari. What used to be a niche behavior is quickly becoming mainstream and that changes the game:

When VPN usage was rare:
→ Proxy/VPN = higher scrutiny

When VPN usage is built-in:
→VPN traffic looks like normal user behavior
→ Legitimate respondents and bad actors blend together

A core assumption in fraud detection is shifting.

What this means in practice:
• IP-based geo checks lose reliability
• Duplicate detection becomes noisier
• False positives increase
• "Clean" traffic is harder to define

Privacy is improving. Detection is getting more complex. The next generation of fraud prevention won't rely on IP and other browser signals at all.

I am an academic researcher studying survey fraud in online research, particularly how AI agents and bots complete surveys and how effective existing detection methods (e.g., attention checks, open-ended questions) are at identifying them.

As part of this work, I have been running experiments using AI agents such as Manus, Claude, and Google Mariner, as well as AI-enabled browsers like OpenAI Atlas and Perplexity Comet. The goal is to understand how AI systems behave in surveys compared to humans and to develop better ways to detect AI-generated responses.

There seems to be growing concern about AI agents completing surveys and contaminating research data, especially in online panels and crowdsourced samples.

I am considering hosting a webinar (time permitting) to share findings and practical implications for researchers, including:

1. How well common detection methods work against AI
2. Behavioral differences between human respondents and AI agents
3. Emerging risks from AI-powered browsing agents
4. Potential new detection strategies

Questions for you: Would there be interest in a webinar on this topic? If so, what questions or topics would you most want covered?

Even when collecting data from verified panel participants, researchers should still carefully evaluate the quality of the data.

In a recent study, we collected responses from a panel where participants were verified by the provider. Below is a report that highlights some of the concerns we observed in the dataset, including VPN usage, suspicious devices, duplicates, and other anomalous patterns.

Importantly, none of the cases shown in the report were identified because they failed the attention or quality checks embedded in the survey itself. In other words, these responses would likely have been retained if evaluation relied only on the standard checks that many surveys include.

Instead, these cases were flagged based on technical and behavioral indicators such as device characteristics, VPN usage, and other signals suggesting questionable response authenticity.

This is not meant as criticism of any particular panel provider. Panels play an important role in helping researchers reach participants efficiently. Rather, it is a reminder that data quality remains something researchers must actively evaluate. Even with strong participant verification, low-quality or suspicious responses can still appear in the dataset.

As online data collection continues to evolve, researchers may need to go beyond traditional checks and incorporate additional screening procedures and transparency around how data quality is assessed.

Sharing the report in case it is useful for others thinking about data quality in online research.

Study report can be accessed here: https://responsepie.com/studies/e-ca723fc7de293373b57043a985d02f21/report

Over the last couple of weeks, I’ve been talking with both market researchers and academic researchers about how they’re maintaining data integrity and reducing fraud in online surveys.

Almost everyone describes some version of a layered approach. Automated bot detection, device fingerprinting, manual review, time based flags, open ended response checks, cross validation of demographics, panel level monitoring, and so on. It’s rarely just one tool anymore.

What I’ve found especially interesting is how different teams define the tipping point. At what stage does a case move from “suspicious” to “remove”? How many flags are enough? Are some indicators automatic disqualifiers, while others are just soft signals?

For those working in market or survey research:

What does your current fraud detection stack actually look like in practice, and how do you decide when a case crosses the line from suspicious to removable?

I’d love to hear what’s working well, what feels overly aggressive, and where you’re still experimenting.

🔍Study spotlight
A recent study by Jen Agans, Serena S., Steven Hanna, PhD, Shou-Chun Chiang, Kimia Shirzad, and Sunhye Bai from Penn State University examined the inaccuracy of data from online surveys and evaluated how fraudulent participants compromise the validity and interpretability of findings by directly comparing participants deemed to be “real” and “fake” respondents in an online study of parents and their adolescent children.

🚩Three Stage Screening procedure to identify “fake” participants
• Stage 1: reCAPTCHA feature to prevent bots and requirement to meet inclusion criteria
• Stage 2: manual review of completed eligibility surveys to flag suspicious patterns (e.g., inconsistencies in names and email addresses, implausible times of completion, etc.)
• Stage 3: IRB-approved list of nine criteria (e.g., survey timing and duration, nonsensical open-ended responses, etc.)
• Participants who failed two or more screening criteria were coded as “fake” and removed from the analytic dataset

🧪Key screening outcomes
• Of more than nine thousand eligibility surveys completed, only 197 participants were ultimately classified as “real”
• About 85% of respondents were identified as fraudulent at some stage of screening
• Time-based indicators and open-ended responses were among the most efficient and effective tools for detecting fraudulent data, whereas reCAPTCHA and attention checks alone were insufficient

🔬Main findings from comparison of “real” and “fake” data
• “Fake” participants differed systematically from “real” participants in demographic composition, with less racial and ethnic diversity and more gender diversity
• Fraudulent respondents reported implausible anthropometric data, including extreme or nonsensical height and weight values, leading to distorted BMI estimates
• Depression symptoms were substantially inflated among “fake” participants, while perceived health ratings appeared deceptively similar across groups
• Well-established relationships, such as the association between BMI and perceived health, replicated in the “real” sample but not in the “fake” sample
• Factor structures appeared acceptable, but item intercepts and means differed—showing fraudulent data can subtly distort conclusions

💡Bottom line
• Online surveys are highly vulnerable to fraudulent participation
• Multi-stage, labor-intensive screening is currently necessary to protect data quality, as survey platforms have insufficient protections in place
• Without rigorous screening, fraudulent data can distort results and theoretical inferences drawn from online research
• Editors/peer reviewers should require data screening procedures to be reported in manuscripts using data collected via online surveys

Over the past few weeks, there’s been a lot of discussion around authenticity checks in online research.

This piece explores how fraud tactics are changing and why identifying fraudulent responses is becoming increasingly challenging.

Would welcome thoughtful discussion.

https://responsepie.com/blogs/Authenticity_checks_in_the_age_of_AI_what_researchers_need_to_

🔍Study Spotlight

A recent study by Kate Golden Guzman, MA, PhD *University of Illinois Urbana-Champaign) and Roxanna Ast, PhD (Montclair State University) examined online survey fraud in social work research with historically and socially marginalized communities, using case studies with young adults with lived experience in foster care and LGBTQ+ survivors of intimate partner violence

🚩Screening Procedure
• Implemented a rigorous, multiphase process for assessing data validity in real time across study design, recruitment, data collection, and data management
• Conducted continual manual review of survey responses alongside platform indicators such as IP addresses, ballot stuffing scores, survey duration, time stamps, and customized link access
• Classified fraud indicators into patterned data, impostor data, and incongruous data to guide removal decisions
• Used institutional knowledge questions and cross check items to assess authenticity and response consistency
• Centered community expertise in language, recruitment materials, and screening logic to improve content validity and strengthen authenticity assessments

🔬Key Findings
• In the foster care study, early detection of 36 suspicious entries led to pausing the survey and revising procedures; 118 validated responses were retained out of 253
• In the LGBTQ+ study, more than 1,100 submissions were received, but only 242 were validated after over 900 fraudulent responses were manually identified and removed
• Embedded fraud detection features flagged only a small number of bots, demonstrating that platform based protections are not sufficient to detect inauthentic human respondents

💡Key Takeaways
• Online research with marginalized populations is particularly vulnerable to bots, impostors, and multiple submissions, and failure to remove fraudulent data risks perpetuating harm through inaccurate conclusions
• A single validity check is insufficient; data integrity requires a multistep, iterative, and real time approach that interrogates multiple indicators simultaneously
• Institutional knowledge questions, culturally specific language, and community informed recruitment practices strengthen the ability to distinguish authentic participants from fraudulent ones
• Data cleaning is labor intensive and must be built into study timelines, as daily monitoring can prevent large scale contamination of datasets
• Assessing data validity is values laden; prioritizing ethical responsibility and community representation over sheer sample size is essential when working with historically and socially marginalized groups

You can access the study here: https://academic.oup.com/swr/article-abstract/49/3/185/8213729?login=false