Although it might be possible to reduce this gap further by increasing the adapter size beyond b = 4096, there might be more efficient ways to approach this problem, including more expressive network architectures for adapters, joint fine-tuning of adapters and global model parameters, etc.
we leave these studies to future work.