Wikipedia examples are longer paragraphs of 66 words on average.
we had initial success sharing the parameters of the two towers which allows training much deeper models without increasing the parameter count.