Don't worry about early downvotes too much. It doesn't show grey based on percentage of downvotes, it shows grey when it goes below 1 point. So if one guy votes it down soon after it's commented, it goes grey. Others usually come along to correct it.