The concatenation of these max values already provides us with a representation of each word as a vector with a fixed length equal to the total number of convolutional ker-nels.
the addition of two highway layers was shown to improve the quality of the language model in (Kim et al., 2016) so we also kept these additional layers in our case.