Depending on the attended visual feature, a formulae is introduced to ascertain whether the current t word is a visual word or not: where c v t is the weight of visual word at time step t, p i t is i-th value in the t-th POS tag representation p t .
the weight of non-visual word is c nv t = 1 − c v t .