The neural system for their experimental analysis is not an attentional model and they argue that attention does not have any impact for learning syntactic information.
performing the same analysis for morphological information, Belinkov et al.