Classically, an LSTM cell is defined as follows (biases excluded for brevity): One way of measuring the complexity of a model is through its total number of parameters.
we take the cell update recurrence parameters w i across LSTMs (we omit the c in the subscript for brevity; the index i runs across the smaller LSTMs) and for any pair we wish the following to be true: To achieve this we pack the vectorized parameters into a matrix: and apply the following regularization term to our final loss: The concatenated output h t is passed through a fully connected layer with bias before being passed through a final softmax layer: To extract a predicted entity typeŷ t at time t, we select the entity type corresponding to the most probable output:ŷ The loss is defined as the sum of the softmax cross-entropy losses along the words in the input sequence.