Remark: In our earlier attempts, we experimented with a self-attention based encoder-decoder with positional encoding similar to (Vaswani et al., 2017) to reduce the encoding time from O(n) (linear) to O(1) (constant) time.
the perfor-mance was inferior to the RNN-based encoder.