People are notorious for their inconsistency. We are easily distractible. We get tired of doing repetitive tasks. We daydream. We misinterpret. So how do we determine whether two observers are being consistent in their observations? You probably should establish inter-rater reliability outside of the context of the measurement in your study. There are two major ways to actually estimate inter-rater reliability.
If your measurement consists of categories — the raters are checking off which category each observation falls in — you can calculate the percent of agreement between the raters. For each observation, the rater could check one of three categories. Imagine that on 86 of the observations the raters checked the same category.
The other major way to estimate inter-rater reliability is appropriate when the measure is a continuous one. There, all you need to do is calculate the correlation between the ratings of the two observers. For instance, they might be rating the overall level of activity in a classroom on a 1-to-7 scale. You could have them give their rating at regular time intervals e.
The correlation between these ratings would give you an estimate of the reliability or consistency between the raters. For instance, I used to work in a psychiatric unit where every morning a nurse had to do a ten-item rating of each patient on the unit.
Although this was not an estimate of reliability, it probably went a long way toward improving the reliability between raters. We estimate test-retest reliability when we administer the same test to the same sample on two different occasions. This approach assumes that there is no substantial change in the construct being measured between the two occasions. The amount of time allowed between measures is critical. We know that if we measure the same thing twice that the correlation between the two observations will depend in part by how much time elapses between the two measurement occasions.
The shorter the time gap, the higher the correlation; the longer the time gap, the lower the correlation. This is because the two observations are related over time — the closer in time we get the more similar the factors that contribute to error. Since this correlation is the test-retest estimate of reliability, you can obtain considerably different estimates depending on the interval.
In parallel forms reliability you first have to create two parallel forms. One way to accomplish this is to create a large set of questions that address the same construct and then randomly divide the questions into two sets. You administer both instruments to the same sample of people. The correlation between the two parallel forms is the estimate of reliability.
One major problem with this approach is that you have to be able to generate lots of items that reflect the same construct. This is often no easy feat.
Furthermore, this approach makes the assumption that the randomly divided halves are parallel or equivalent. If the observers are given clear and concise instructions about how to rate or estimate behavior, this increases the interobserver reliability. Experience is also a great teacher; researchers who have worked together for a long time will be fully aware of each other's strengths, and will be surprisingly similar in their observations.
Auerbach, C. Statistical Methods for Estimates of Interrater Reliability. In Roberts, A. Jackson, S. Rubin, A. Martyn Shuttleworth Aug 16, Interrater Reliability. Retrieved Nov 11, from Explorable. The text in this article is licensed under the Creative Commons-License Attribution 4. That is it. You can use it freely with some kind of link , and we're also okay with people reprinting in publications like books, blogs, newsletters, course-material, papers, wikipedia and presentations with clear attribution.
Menu Search. Menu Search Login Sign Up. You must have JavaScript enabled to use this form. Sign up Forgot password. Leave this field blank :. Cart mytutor2u mytutor2u. Psychology Explore Psychology Search Go. Psychology Topics. Overview Latest Topics Watch now. From the Reference Library.
0コメント