Most often, the creation of these out-of-domain treebanks follows the existing annotation guidelines created by the creators of the newspaper treebank, which would -at least in principle -make it possible to evaluate the generalization performance both of purelysupervised and domain-adapted parsers on these new datasets.
some caution is in order, as in many cases, annotation guidelines or even informal practices guiding the annotation deviate.In the case of Dredze et al.