The most straightforward way to investigate this problem would be to ask one or more annotators to re-rank our dictionaries, and compute the inter-annotator agreement.
besides the difficulty and time-consuming nature of the task, any simple metric derived from such a process would provide information that was useful only in the context of the absolute values of our −5 to +5 scale.