The comparison here suggests that LSTM and self-attention neural networks are able to capture better implicit structures than handcrafted features.
one idea to resolve this issue is to design an alternative mechanism to capture such useful structural information that resides in the input space.