We present experiments affirming that this gain is due to an explicit composition operator rather than the presence of predicted syntactic annotations.
training was done using a language modeling objective that predicts the next word given the prefix; at test time we compute agreement error rates by comparing the probability of the correct verb form with the incorrect one.