Do you ever find yourself questioning how accurate your survey results are? You’re not the only one. At quantilope, we know the value survey insights can provide brands and make sure that all clients feel confident with the results their online surveys will provide them before going live.
Read on to explore some of the common doubts people have toward questionnaires, and how to overcome those reservations to obtain and leverage valuable consumer insights.
Table of Contents:
- How common is lying on surveys?
- Reasons people lie on surveys
- Tips for getting honest answers on surveys
- Dealing with dishonest survey answers
- Conclusion
How common is lying on surveys?
While we’d like to convince you that all survey respondents tell the truth 100% of the time, we know that’s simply not realistic. Surveys have low stakes for many respondents and they may limit the effort they exert when aiming to get their earned incentive (be it cash, gift cards, points, etc.).
The good news? A study from 2021 found that 75% do not lie much (‘much’ meaning 0-2 lies a day) and only 8.9% lie to strangers (of which survey providers might be considered). Compare that to 51% who lie to friends and 21% who lie to family, and the survey lying likelihood is looking pretty good.
The even better news? Advanced survey platforms have the power to spot these survey liars instantly through data cleaning - more on this in later paragraphs.
Reasons people lie on surveys
So we know some respondents are going to lie on survey questions... but why?
There are a number of reasons why survey respondents might lie in place of giving honest answers. Below are just a few commonly observed instances:
Survey biases
Survey biases are phenomenons that describe the many ways a survey itself might contribute to respondents lying.
For example, there’s acquiescence bias - which is when respondents choose to agree with something even if it doesn’t match their true feelings. Respondents will do this if they think there is a ‘right’ answer by agreeing with something, or that agreeing with something reflects more positively on them as an individual (i.e. ‘We need to cut back on plastic use to save the planet’ and saying what they think they “should” say, rather than what they truly think).
Then there’s social desirability bias, which is similar to acquiescence bias in that respondents choose what is viewed more favorably, but in this case, it’s not as limited to an agreement scale. With this form of bias, respondents might lie about how much income they earn, their relationship status, how often they drink/smoke, or how often they exercise at the gym. Because there’s an anonymity factor to online surveys, respondents can manifest the best version of themselves to be seen as ‘desirable’ and project who they may want to be, rather than who they truly are.
There are several other forms of survey bias that can lead to a respondent’s tendency to lie in a questionnaire, but fortunately, there are ways to avoid them as well.
Wanting to be polite
Believe it or not, some survey liars may do it out of politeness rather than for personal gain. Say a survey is asking about a shopper’s in-store experience; the shopper may have had a horrible time shopping because the store temperature was too hot, they couldn’t initially find what they were looking for, or something they wanted was out of stock. However, they were later assisted by a really friendly store associate and they know the survey results may somehow be linked back to that individual, so, they answer more positively than they otherwise would have to hopefully avoid negative repercussions for that store associate.
Or, say a survey is asking for employees' satisfaction in the workplace. Though these surveys are typically (and should be) anonymous, an employee may fear that their responses will be tied to their name (despite the assurance that they are not), so they answer more politely than they truly feel.
To intentionally skew results
In contrast to the above, there are going to be some survey respondents who lie intentionally to skew the survey data - whether just for fun or for a motive.
Say for example that a respondent gets to the demographic section of a survey and they believe that if they select a certain gender or race, they will be classified into a certain group. This respondent may choose a different identification on purpose to avoid such classification.
Another example might be if there is a brand-related question and a respondent wants to attempt to inflate the metrics for the brand they really love. To do so, they may say they ‘hate’ all other brands except for their favorite, when really, they might like a number of brands listed.
To get an incentive
Lastly, and as alluded to in the earlier paragraphs, many respondents may take part in surveys just to get the end result: the incentive. They don’t really put much time or thought into their answers because they only care about the prize in the end and will move as quickly as possible to get there. While this isn’t true of all respondents (there are many that actually enjoy providing their feedback and feeling part of a brand’s operations), it’s something to account for when reviewing your survey data.
Those who are racing through a survey, selecting the same response at every question, or are providing gibberish/poor quality answers at open-ended questions are likely just in it for the incentive, and brands can clean these individuals out of their final data set.
Back to Table of Contents
Tips for getting honest answers on surveys
At this point, you might be stressing about your survey validity or questioning why you’d ever run an online survey again. Don’t worry - we have you covered!
Below are some tips and tricks for getting honest responses from survey respondents so you can feel confident in your insights every time.
1. Make surveys anonymous
Making it clear that surveys are anonymous can help reduce the inclination to lie by not attaching a respondent’s name, identity, or any other personal data from demographic questions to them personally. Start your survey with an opening message emphasizing that all data captured is kept anonymous and data will only be reviewed in aggregate. When respondents see this disclaimer, they are less likely to feel they have to lie to be ‘accepted’ or ‘socially desirable’, and response rates will be higher.
If a surveyor can’t make their survey anonymous, at a minimum they can provide an ‘opt-in’ selection so that only respondents who are willing to share their information will proceed through the survey. Similarly, for questions that ask for personal information, having a “prefer not to answer” option will give these respondents a choice not to participate, making them less likely to lie.
2. Avoid leading questions
Thinking back to the acquiescence bias mentioned above, when respondents feel it’s better to agree with something, they will do so regardless of their true sentiments. To avoid this, make sure your questions aren’t ‘leading’ them toward any given answer.
This means removing prompts from a question that might influence how a respondent would react. For example, the below are not good survey questions to ask, given their leading tone:
-
How much do you agree that we need to cut down on plastic use to save the planet?
-
To what extent do you agree that going to the gym is the best thing you can do for your health?
These types of questions can easily be rephrased as the following to cut out the ‘leading’ prompts:
-
Do you agree or disagree that we should change our plastic usage?
-
Which of the following do you consider the best thing one can do for their health? (go to the gym, eat healthy meals, drink water, get good sleep, etc.)
3. Consider your survey structure
One of the simplest ways to avoid lying in a web survey is to keep the survey design short and to let respondents know how much time they have left. It might sound silly, but when respondents know how much more time they have left to commit to a questionnaire, they are less likely to speed through the answers just to reach the end (not knowing when that might be, it could seem never-ending).
Aside from the length of the survey, consider diversifying the structure of your questions to keep things interesting for respondents and to ensure they can’t just select the first option in a multiple-choice question every time. Mix it up with single-select questions, multi-select questions, matrices, open-ended answers, a gamified methodology like MaxDiff and Price Sensitivity Meter, and maybe even video responses.
In fact, one of the best ways to capture honest feedback and immediate gut reactions is through the use of an advanced method known as the implicit association test:
4. Add redundant questions
Adding slightly altered variations of the same question can help see if someone is lying if they were to give completely different answers to each one. Say for example a survey first asks ‘Where do you live?’ and provides the options: ‘suburbia’, ‘large city’, and ‘rural town’. A few questions later, a new question reads ‘Which of the following are true?’: ‘I live in the suburbs’, ‘I live in a big city’, and ‘I live in a rural town’. If a respondent says ‘suburbia’ in the first question but then says they ‘live in a big city’ in the second question, this respondent is probably not giving honest answers to your questions.
This is a similar tactic in surveying known as a ‘red herring’ - which is when a researcher includes a fake or irrelevant answer to a question to catch respondents who aren’t giving truthful answers. For example, if a survey asks ‘Which of the following soda flavors are your favorite’ and includes an answer option ‘soccer’ - any respondent who selects this answer will be flagged as invalid as it’s clear they’re not thoughtfully reading the questions and answer options.
5. Use behavioral questioning techniques
Using behavioral instead of attitudinal questions can lead to more honest answers because respondents are more likely to be honest about what they do than how they feel. For example, you may be more honest about how often you go to the gym each week vs. how important it is to go to the gym each week. Or, you might find it easier to give a truthful answer about which brands you actually buy than which brands you simply like.
Back to Table of Contents
Dealing with dishonest survey answers
As promised earlier, this section explores some of the ways to catch those pesky survey liars in the act. You’ll need a way to clean your data - whether that’s an automated tool on a survey platform or a dedicated data cleaning team.
Through data cleaning you can identify the following types of respondents that are most likely not being truthful throughout your questionnaire:
-
Speeders: These are respondents who finish a survey in its entirety faster than a time that would be considered reasonable. This could be determined through a percentage threshold (i.e. anyone who completed the survey faster than x% of respondents will be cleaned out), or by a number (i.e. anyone who completed the survey in less than 3 minutes is cleaned out). How to implement speeding cleanouts should be determined by the researcher, as one rule of thumb does not always fit — based on logic and qualifications, two survey respondents could have very different survey experiences, which could mean that they have two different lengths of surveys that are both reasonable.
-
Straight-liners: These are survey takers who breeze through a survey and select the same response every time (i.e. ‘agree’ with all statements), typically at scaled questions. Because there are cases where it could be valid that a respondent would agree with every statement or select the first statement at every question, straight-liners should be looked at closely depending on the question and cross-checked to see if they are flagged for any other data cleaning filters as well.
-
Poor quality responders: You can use open end answers to identify respondents who may not have been taking the survey seriously. These respondents will likely input text responses that 1.) are gibberish and have no meaning (i.e. a string of text that makes no sense), 2.) that are short/non-relevant answers such as ‘very good’ or ‘like very much,’ or 3.) that make sense textually but have no relation to the question asked (i.e. the response describes a favorite food when the question asked about preferred electronic brands).
These types of respondents are all ones to clean out of a survey using their identifier (usually a respondent ID number). Once cleaning these respondents out, you may have to return to field to collect data from additional respondents to meet your final quota, but that’s all part of a standard market research survey process to obtain high-quality, accurate answers.
Back to Table of Contents
Conclusion
For truthful answers from your survey results, it’s important to plan ahead in terms of your survey design as well as how you will clean out poor-quality respondents.
Designing your survey to be short and avoiding leading or sensitive questions are two of the simplest ways to increase survey response rates. Respondents want to be aware of their time commitment as they take a survey, and are more likely to answer honestly when they know their personal data isn’t attached to their feedback. However, survey liars are sometimes inevitable and you need a plan on how to identify these participants and clean them out of your total sample.
quantilope’s automated technology includes automatic data cleaning so you never have to worry about digging through a data set manually and finding respondents who sped through a survey or answered every question repetitively.
To spend less time cleaning out participants and more time putting together actionable insights, get in touch with quantilope below: