The ultimate test would be to use predicted compositionality scores in downstream applications and tasks involving some degree of semantic processing, ranging from MWE identification to parsing, and word-sense disambiguation.
it would be interesting to predict compositionality in context, in order to distinguish idiomatic from literal usages in sentences.