I think its because we've eveolved to be more sensitive to faces and pay more attention to them. I suspect the whole image is off, but we really notice it in faces and particularly the eyes.