To our best knowledge, there exists no largescale empirical comparison of different activations across a variety of tasks and network architectures, and even less so within natural language processing (NLP).
1 the question which activation function really performs best and most stably across different NLP tasks and popular NLP models remains unanswered to this date.