Note that many existing generator architectures (Miyato and Koyama, 2018; Wang et al., 2018; Karras et al., 2019, 2020) have residual blocks starting from higher dimensions (eg. 512, 1024) in lowresolution then gradually decrease the dimension as feature maps are spatial upsampled.
we use the same vocabulary used in BERT 4 and LXMERT with size 30522.