In a standard "improvement over baseline"-setup, this would be problematic.
the results should not be interpreted with respect to their absolute value on the respective tasks, but with respect to the relative differences.