Metrics p-value vv+uc > vv All < 10 −7 vv+uc > uc All < 10 −20 uc > bilinear All except P@5 < 0.006 bilinear > svm All < 10 −20 vv > svm All < 10 −20 svm > nb All < 10 −8 nb > cos All < 10 −20 Not surprisingly, vv performs poorly for raters or authors with no ratings observed in the training data.
once we have a small amount of ratings, it starts to outperform uc, even though intuitively, the textual information in the comment should be more informative than the authorship information alone.