Inter-rater reliability is a measure of consistency used to assess the degree to which different judges (or raters) agree in their evaluation (or scoring) decisions of the same phenomenon. Inter-rater reliability is high when reviewers demonstrate that they consistently reach the same or very similar decisions. A formal training and calibration procedure is usually needed to achieve this result, and the calibration involves calculating reliability coefficients.