In our experiments, the batch approach helps to save 41% of the processing time when comparing with the asynchronous way.
more than 75% of questions take less than 150ms with rerank and less than 200ms with generation.