Using Requests and lxml is a better solution — except when you need many concurrent spiders, in which case you should be using Scrapy (you'd probably be looking for the Middleware discussed here, too: https://groups.google.com/d/msg/scrapy-users/WqMLnKbA43I/B3N...).