At run-time, the input embedding and the output embedding are binarized matrices.
at train-time, float versions of the embeddings, which are used for calculating the binarized version of embeddings, are still maintained.