After many experimentations, a general "shallow and broad" configuration was adopted.
simplified grapheme bi-grams are first embedded into 128-dimensional vectors 3 and then encoded in one layer of a bi-directional LSTM, resulting in a final representation consisting of a 512-dimensional vector that concatenates two 256-dimensional vectors from the two directions.