The evaluations are on "how well algorithms submitted by professional data scientists and amateur statistics wizards could predict the scores assigned by human graders." As long as some humans are always included as a control, this shouldn't be too big of a problem.