The former is referred to as world model learning, and the latter direct reinforcement learning.
user simulators usually lack the conversational complexity of human interlocutors, and the trained agent is inevitably affected by biases in the design of the simulator.