Multiple raters

Very often in usability research, an investigator will want to get an idea of how things are rated. Perhaps the investigator has been tasked with assessing the subjective readability of ten pages on a website and gets five people to offer ratings on a 5-point Likert scale. How then can he or she see whether the ratings are the same or not?

The best way to look at this is to use statistical analysis. Good analysis will allow the investigator to easily say whether there is a significant difference or not. However, I have commonly encountered people using correlations assuming that if all are significantly associated, then the ratings are the same. This is certainly a possible solution, but it’s tricky: using the above example, our investigator will have to perform ten different calculations: if any result is not statistically significant, then the ratings are not the same. In addition, by subjecting the same data to several, similar analyses, the investigator might be causing alpha inflation.. Because every analysis has a probability of one in twenty of happening due to chance, repeating analysis reduces this. With ten different tests, the probability of getting a significant result drops to one in two. The investigator would be making a serious error in doing this.

But fear not! There are good tests that can be used to test the consistency of several raters in just one go. These are known as tests of inter-rater reliability.