We examine the quality of the MR extraction with a qualitative study evaluating YELPNLG MR to NL 1 2 3 4 5 6 7 8 9 1 0 1 1 pairs on various dimensions.
we evaluate content preservation (how much of the MR content appears in the NL, specifically, nouns and their corresponding adjectives from our parses), fluency (how "natural sounding" the NL is, aiming for both grammatical errors and general fluency), and sentiment (what the perceived sentiment of the NL is).