A possibility is that models do encode information in the entity representations, but it doesn't get used in this task because of how the utterance is encoded in the hidden layer, or that results are due to some quirk in the specific setup of the task.
we replicated the results in a setup that does not encode whole utterances but works with single attributes and relations.