To address this issue, we introduce a knowledge sentinel that records the information of the current context and use a mixture model to allow for better tradeoff between the impact of background knowledge and information from the context.
we compute a sentinel vector s t as: where W b and U b are weight parameters to be learned.