In this paper, when combining word-pair distances learned by different components of the contrasting inference layer, we use some top hidden layer(s) to provide a non-linear combination.
we use two hidden layers, which is able to express complicated functions (Bishop, 2006).