Technically, rather than comparing M i .v t−1 with M i .g and adopting operations accordingly, we propose a soft way to update the state memory panel with SUB and ADD, since (1) it is difficult to identify over-expression or sub-expression by comparing two distributed representations; and (2) the hard way will break differentiablility of the model.
we define g t ∈ R d×l as a gate to control the use of SUB or ADD where g be the changes from the SUB operation and the ADD operation respectively, then M i .v t−1 is updated aŝ where • means element-wise multiplication, and g t (i), ∆ SU B t (i), and ∆ ADD t (i) can be defined as and where ⊕ is a concatenation operator.