However, using observed agreement to measure reliability does not take into account agreement that is due to chance and hence is not a good measure of reliability.
an analysis using Cohen's Kappa statistic (Cohen, 1960)    chance agreement would equate to 0.