Experiments on non-expert dependency annotation from scratch were previously reported for French, suggesting low agreement rates (79%) with an expert annotation benchmark (Gerdes, 2013).
most such evaluations were conducted us-ing annotation setups that can be affected by an anchoring bias (Carroll et al., 1999;Rambow et al., 2002;.