We also trained a model which combines both sets of features, but we found that we get better performance by training the two models separately, then averaging the models by computing the respective averages of their features' weights.
we use the model-averaged version of the models that consider context in the test set experiments.