One one hand, Fisher kernels are derived from information-geometric arguments (Jaakkola and Haussler, 1999) which require that the kernel reduces to an inner-product of Fisher scores.
polynomial and RBF kernels often display better performance than a simple dot-product.