This use of the metrics would consider agreement on categories such as "tokens whose head is token number 24", which is obviously not a linguistically informative category.
we have to reject this way of assessing the reliability of dependency syntax annotation.