How To Improve Research Sample Quality

Maintaining high-quality data can make the difference between selecting a successful advertising campaign or product concept and one that flops. Market research panel providers take great steps to ensure the respondents in their panels are vetted vigorously, but unfortunately, that doesn’t ensure that every complete for your survey will be of a quality worth keeping.

This post will walk you through how you can improve the quality of your research sample and identify respondents that hinder the quality of your data.

Table of Contents: 

• What is sampling research? 
• Advantages of sampling in research 
• The impact of poor sample selection 
• How to improve sample quality for research 
• Quality research starts with a quality sample 
  
  

What is sampling research? 

In survey research, sampling is the process of collecting willing survey respondents to answer research questions. Businesses then use respondents' answers to craft actionable insights for any outstanding business questions.

Survey respondents can come from a variety of sources, but most often, teams will work through a survey panel provider. These panels are massive collections of consumers that have agreed to take surveys in exchange for incentives (i.e. money, gift cards, reward points, etc.). Panel providers have ways of sourcing particular demographics and monitoring who the survey is sent out to depending on necessary quotas. Aside from panel providers, another common form of data collection is done by sending your survey to an existing customer list - for example, email subscribers.

Back to Table of Contents

Advantages of sampling in research

Sampling is an important step of data collection, as you'll want the right people to complete your questionnaire for high-quality data. Working through a consumer panel is often the most practical way to collect data for your survey — as it’s typically more affordable and faster than other data collection methods. For those working with a limited budget or who have a deadline for results, consumer panel sampling is a great option to consider.

Back to Table of Contents

The impact of poor sample selection 

Without taking measures to ensure you're collecting the right people for your survey (by working with a panel provider or closely monitoring the quality of your completes), your final data set can include inflated (or deflated) metrics that don’t accurately represent the sample you’re after.

More than simply impacting percentages, poor-quality responses can lead to ill-informed business decision-making. For example, you might make a product decision (i.e packaging, scent choice, etc.) based on feedback from respondents that don't accurately reflect the target audience buying your products. The result is wasted time, budget, and resources, along with a product on the shelf that doesn't sell. 

Back to Table of Contents

How to improve sample quality for research

Regardless of how you collect your sample, remember you have final control over the quality of your data set. 

While some ‘bad’ respondents may make it into your data set, you’re the one to control who makes the final cut. Below are a few steps to ensure a high-quality sample for your survey data:

1) Learn about your panel’s processes:
You always have the right to ask panels what steps they take to ensure data quality. This will help you learn about the diligence that the company puts into its sampling process to understand where its participants are coming from and what criteria they set for qualification. If this request is foreign to the panel provider, it may not be a panel you want to work with.

2) Include trap questions:
Somewhere in the screener of your questionnaire, you can include quality check questions that will look to ‘trap’ or ‘trick’ respondents that could be bots or respondents that are not putting care into their responses. One example of a trap question that you’ve probably seen before is an image of squares, asking respondents to select all of one item, such as ‘dogs’, ‘bikes’, or ‘mountains’. Another way to trap poor-quality respondents is by asking them to answer small functionality tasks such as ‘8+3’, or saying “select the third answer to continue”. As you can tell, these questions are not related to the content of your survey and are really in place to conserve the quality of data you achieve.

3) Review and clean the data:
Cleaning your data is a must. Before feeling fully confident in your analysis, you want to make sure you have valid answers by conducting quality checks. When reviewing a data file, there are typically three things you want to keep an eye out for:

The first is people taking your survey too quickly. Taking a survey too quickly often means that respondents are not thoughtfully answering the questions (these are known as ‘speeders’).

The second thing to look for is straight-lining, meaning people that select the same scale point on every scaled survey question, over and over. This could be something like selecting ‘strongly agree’ to all statements or selecting the last answer in a multiple choice at every question. While those answers might be valid and a coincidence, it’s important to review such cases to ensure they make sense. For example, if a respondent selects they ‘strongly agree’ that coffee is delicious, and at the next question they ‘strongly agree’ that coffee is disgusting, that respondent is not thoughtfully answering your questions.

The last item to be wary of is poor-quality, open-text feedback. This could be jibberish responses (i.e. askswesas) or responses that are unrelated to the question being asked, such as ‘very good’ to a question asking to list out brand names.

Often, respondents will fall into several of these data-cleaning buckets — if you’re speeding through the survey, you’re likely also clicking the same scale points and typing in irrelevant answers at open ends. It’s still a good idea to review each of these buckets independently, though comparing each of these elements for one respondent might help make a decision for those you’re unsure about.

Design your studies to get quality answers from respondents

Your survey design will also play a large factor in helping you avoid poor data quality. Exhausting respondents can lead to them being more careless as they reach the later parts of your survey.

Survey exhaustion/fatigue can stem from the length of the survey but also the types of questions being asked. For example, your survey might not contain an extraordinary number of questions, but if it contains many questions that require intense thought, that may lead to fatigue. This is something to keep in mind when deciding how many advanced methods to use in a single survey, how many open-text questions you ask of your respondents, and the number of options in multiple-choice or ranking questions. Also remember that the more detailed your survey design is, the more information you’ll have to squeeze into a final dashboard or presentation. 
Back to Table of Contents

Quality research starts with a quality sample

As a reminder, you are fully in control of your final data set. Using the recommendations outlined above, you have the power to hinder poor-quality respondents from completing your survey, and remove those that happened to sneak through.

With automated research platforms like quantilope, leverage tools such as survey templates and automated data cleaning for the highest quality survey data possible.