This is likely due to the fact that SM was trained on the SALICON dataset; a higher correlation can probably be achieved by fine-tuning the salience model on the PASCAL Actions Fixation data.
(2016) report for their question answering ask, indicating that verb classification is a task that can be performed more reliably than Das et al.