One exception is that we weren't able to reach acceptable agreement on a feature of NPs often claimed to affect ranking, thematic roles (Sidner 1979;Cote 1988;Stevenson, Crawley, and Kleinman 1994); the agreement value in this case was κ = .35.
we were not able to evaluate ranking functions based on thematic roles.