Clean Your Data

About Scrubbing Data

Identifying and scrubbing faulty data is critical to achieving an accurate result. The Check Data page helps you find bad data and exclude it from the results of your expairiment.

Why Check for Bad Data?

For a number of reasons:

  • You took your own survey. You want to QA your survey, which you do by taking it, however you may not want your responses analyzed along with other participants.
  • You forgot to close the survey, someone later stumbles upon it and takes it, polluting your results
  • A survey participant is distracted and gives you what looks like random responses.
  • A survey participant abandoned the survey and didn’t give enough meaningful responses.
  • Someone is gaming your survey so they can skew the results toward a result that they favor.

Pairlab lets you find these situations so you can decide whether to keep responses for a participant or not.

How to Check Your Data

Use the Check Data page, which has a table with these columns:

  • Checkbox -- Click the checkbox for a participant you wish to exclude or include from the results. Click the Toggle Analyze button at the bottom of the table to toggle between excluding and including them in results.
  • No. -- An ordinal number assigned to the participant to help you understand the data.
  • ID -- Part of the unique ID number assigned to the participant shown to help you understand the data.
  • Voted on -- The date and time the participant cast their last vote.
  • Analyze -- "Yes" (green dot) means the participant is included in the results, "No" (gray dot) means they are excluded from the results. You may toggle a participant's Analyze status by selecting the checkbox to the left and clicking the Toggle Analyze button at the bottom of the table.
  • Stanley Score -- The degree of randomness in the participant's vote. A score below 25 (green) is generally acceptable, a score between 25 and 34 (orange) indicates moderate randomness, and a score of 35 or above (red) may be unacceptable. A missing score means there weren't enough responses calculate a score, and the "responses" flag is thrown as a result.
  • Flags -- Comments that warn you about problematic responses. "Skips" means the participant clicked the "Skip these two ideas" button numerous times in a row. This indicates either the participant struggled to understand how the ideas compare or they tried to game the survey to favor one idea over others. "Responses" indicates there were few responses for a participant. Too few responses may only be a problem when the number of votes does not exceed the number of pairs, as shown on the Results Summary.

pairLab determines the Stanley Score and Flags a few hours after the participant voted to be sure their session ended. An asterisk indicates the score and flags are pending.

How to Exclude or Include Participant Data

You’ve looked over your data and decided which responses you want to exclude, or you changed your mind and want to include responses you previously excluded. How do you do that?

  • Click one or more check boxes for the participants you want to exclude
  • Scroll to the bottom of the table
  • Click Toggle Analysis.

This action will switch the Analyze indicator from its current state to the opposite. Excluded participants will not have their responses included in results, included ones will.

Posted by Christian
on August 30, 2020