Note that the human studies and automatic evaluation are complementary to each other: while MTurk annotators are good at judging how natural and coherent a response is, they are usually not experts in the Ubuntu operating system's technical details.
automatic evaluation focuses more on the technical side (i.e., whether key activities or entities are present in the response).