> *but you'll have a non-zero rate of pages you think you've downloaded but ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		pooriaazimi on Aug 10, 2012 \| parent \| context \| favorite \| on: How to crawl a quarter billion webpages in 40 hour... > but you'll have a non-zero rate of pages you think you've downloaded but you actually haven't, depending how you tune it. My (not-very fresh) memory of what bloom filter actually is tells me that this "non-zero rate" you're talking about must be HUUUUGE. In order of millions of pages. Am I right?

cypherpunks01 on Aug 10, 2012 | [–]

You're right if OP was using a small bloom filter, and wrong if it was a big one. Hence, the phrase, "depending how you tune it."

JoshTriplett on Aug 10, 2012 | [–]

You can construct a bloom filter with an arbitrarily low error rate.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact