If you're hitting HN's access thresholds, you might consider incrementally parsing data from http://hnstream.com

That may help with analyzing comment content at least. Not so much with news rankings though.

Thanks. hnstream is really cool.

You're right. I have limited frequency of crawling due to HN's access thresholds.

hnstream will help me. I will try it.