In English, for example, The Queen paid a visit to the Prime Minister + a visit of the Lady to the Prime Minister →*The Queen paid a visit of the Lady to the Prime Minister.
the visitor cannot be a modifier of visit.