By the linearity of expectation, we have that , and so the mean across w mean will also be close to zero at least for small k. In practice, this seems to hold even for moderate k in naturally occurring sentences, as seen in Figure 1.
as embeddings are ultimately just arrays of numbers, we are free to take alternative viewpoints other than the geometric ones, if they lead to illuminating insights or strong-performing methods.