Simianer, Riezler, and Dyer (2012) propose another method for mixing parameters that, instead of averaging at each iteration, chooses to preserve only the parameters that have been learned over all shards, and sets all the remaining parameters to zero, allowing for a simple sort of feature selection.
we define a S × M matrix that combines the parametersw (t+1) s at each shard asw (t+1) = w (t+1) S , takes the L 2 norm of each matrix column, and averages the columns with high norm values while setting the rest to zero.