This implementation has the advantage of simplicity as it requires minimal changes to the training and deployment code, but we note that a more complex implementation utilizing sparse matrices and sparse matrix multiplication could potentially yield speed improvements.
this may seem surprising, as we might not expect a sparse model to significantly out-perform a model with five times as many parameters.