This is because when training the ELMO-C model reported in 4.2, we actually train a forward ELMO-C on two cards and train a backward ELMO-C on two other cards, which reduces the communication cost by half.
a pre-trained word embedding, which projects words with different meanings into different positions in space, is a natural choice for the projection matrix W and can help preserve much of the variance of A.