Clean Your Data

About Scrubbing Data

Identifying and scrubbing faulty data is critical to achieving an accurate result. pairLab gives you several tools to help you find bad data. The Check Data table shows you information for each participant in your expairiment.

The pairLab Check Data table
The pairLab Check Data table

The fields are:

  • Checkbox -- Click the checkbox for a participant you wish to exclude or include from the results. Click the Toggle Analyze button at the bottom of the table to toggle between excluding and including them in results.
  • No. -- An ordinal number assigned to the participant to help you understand the data.
  • ID -- Part of the unique ID number assigned to the participant shown to help you understand the data.
  • Voted on -- The date and time the participant cast their last vote.
  • Analyze -- "Yes" (green dot) means the participant is included in the results, "No" (gray dot) means they are excluded from the results. You may toggle a participant's Analyze status by selecting the checkbox to the left and clicking the Toggle Analyze button at the bottom of the table.
  • Chimp Score -- The degree of randomness in the participant's vote. A score below 25 (green) is generally acceptable, a score between 25 and 34 (orange) indicates moderate randomness, and a score of 35 or above (red) may be unacceptable. A missing score means there weren't enough responses calculate a score, and the "responses" flag is thrown as a result.
  • Flags -- Comments that warn you about problematic responses. "Skips" means the participant clicked the "Skip these two ideas" button numerous times in a row. This indicates either the participant struggled to understand how the ideas compare or they tried to game the survey to favor one idea over others. "Responses" indicates there were few responses for a participant. Too few responses may only be a problem when the number of votes does not exceed the number of pairs, as shown on the Results Summary.

pairLab determines the Chimp Score and Flags a few hours after the participant voted to be sure their session ended. An asterisk indicates the score and flags are pending.

Best Practices

What Is the Chimp Score?

The Chimp Score is named in honor of the most difficult player to beat in Chessmaster 9000: "Stanley," a chimpanzee that made chess moves randomly (the game against Stanley almost always ended in a draw). The Chimp Score measures the degree of randomness in the participant's vote. Nearly every participant will show some degree of randomness in their responses. Randomness becomes a problem as it increases.

The color indicators are meant to guide you. You must decide how much randomness you allow. The score's minimum value is 0 and its maximum value is 100. Surveys with larger idea sets tend to have higher chimp scores.

Be the Customer (Once)

Take your own survey, it can help you refine it for good results. After you're done, exclude yourself from the dataset.

Posted by Christian
on August 30, 2020