We propose to empirically validate the importance of information aggregation in multihead attention, by comparing the performance of the standard linear function and advanced aggregation functions on various tasks.
we cast information aggregation as the assigningparts-to-wholes problem (Hinton et al., 2011), and investigate the effectiveness of the routing-byagreement algorithm -an appealing alternative to solving this problem (Sabour et al., 2017;Hinton et al., 2018).