We focus on the queries from the 21 submissions that are considered incorrect by ESM but correct by our test suite evaluation, randomly sampled and manually examined 100 of them.
the current Spider metric leads to a 2.5% false negative rate on average and 8.1% in the worst case, indicating that test suite accuracy is needed.