We conduct subjective evaluations by collecting annotations about the qualities of the
translated songs from music school students.
Following mean opinion score (MOS) in speech synthesis, we use five-point Likert scales
(1 for bad and 5 for excellent). And we evaluate the songs on four dimensions:
sense, fidelity to the meaning of the source lyric;
style, whether the translated lyric resembles song-text
style;
listenability, whether the translated lyric sounds melodious with the given melody;
intelligibility, whether the audience can easily comprehend the translated lyrics if sung with provided melody.
The last two dimensions require the annotators to sing the song.
Instructions
The instructions we sent to each annotators are shown as below: