This looks really interesting, I'm going to bookmark it!
> Proxy IP address rotation
I wonder where you would get a decent proxy list nowadays. In the late 90s, you could find list of accidental open proxies that "hackers" collected. But nowadays I've only seen really shady "residential proxies" that are basically private PCs with malware. Is there a decent source for proxies that are not widely blocked and not criminal and not too expensive?
And by the way, while many people here question the morality of scraping, I have at least two legitimate use cases:
1) I built a little "vertical" search engine for a niche, using a combination of scraping and RSS. It doesn't consume much traffic and does really help people discover content in this niche.
2) A friend does research on urban development and rent prices and they scrape rent ads to get their data (or let students type it off manually, I'm not sure...)
I run a number of proxies for (legally) scraping government data sites with overly restrictive views/hour policies.
They're just squid boxes spun up on various cheap VPS providers. A $5 digitalocean instance will power a lot of squid traffic, and as long as you stay away from AWS, the IP ranges tend not to be banned.
Just make sure to lock down your allowed source list, so you don't become one of those hacked open proxies!
> Proxy IP address rotation
I wonder where you would get a decent proxy list nowadays. In the late 90s, you could find list of accidental open proxies that "hackers" collected. But nowadays I've only seen really shady "residential proxies" that are basically private PCs with malware. Is there a decent source for proxies that are not widely blocked and not criminal and not too expensive?
And by the way, while many people here question the morality of scraping, I have at least two legitimate use cases:
1) I built a little "vertical" search engine for a niche, using a combination of scraping and RSS. It doesn't consume much traffic and does really help people discover content in this niche.
2) A friend does research on urban development and rent prices and they scrape rent ads to get their data (or let students type it off manually, I'm not sure...)