Freezing weights learned in unsupervised training ( Figure 2) performs better than fine-tuning them; also, Table 3 shows that the best result is achieved if no convolution is used in mfCNN.
the best configuration for paraphrase identification is to "forward" fixed-size interaction matrices as input to the logistic regression, without any intermediate convolution layers.