I had a friend in college who never thought he was good at math. Then one day in 10th grade he started going through math books. He is now an extremely talented mathematician at Harvard. Insert lockhart's lament, I suppose.
Part of the problem I think is that we assume that people who don't get math early on are just "bad at it" as opposed to people who don't get reading. If a student doesn't learn to read, we think "how can we better teach them? Do they have dyslexia? What are ways we can teach dyslexic children to read? We don't ask the same questions when a student doesn't get math, we just say "oh, well, they suck at that then."
I find it amazing how narrow the gap is between brilliant and average, particularly for intelligent people. A couple of hours of reading a textbook can sometimes be equivalent with about a year of public school education.
Case in point: Took AP Computer Science my sophomore year of high school. I got C's on most of the tests (mostly code tracing) for the first two months or so. After getting a particularly bad grade on a test (near failing, and I got all A's in everything else), I went home and thought "fuck this" and pulled out some Java book and read it for about 5 hours. After this, I moved from being about 40th percentile to best student in my class. I got high A's the rest of the year without trying particularly hard, and I actually enjoyed the class, and this lead to me starting to program on my own for fun, and eventually becoming a pretty good programmer.
Part of the problem I think is that we assume that people who don't get math early on are just "bad at it" as opposed to people who don't get reading. If a student doesn't learn to read, we think "how can we better teach them? Do they have dyslexia? What are ways we can teach dyslexic children to read? We don't ask the same questions when a student doesn't get math, we just say "oh, well, they suck at that then."