Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

An earlier ruling actually ordered LinkedIn to stop attempting to block the scraping using technological measures, too.


I believe that LinkedIn were enjoined from using measures to limit HiQ specifically from scraping their site, not from general authorization-based measures that might have the corollary effect of limiting access by HiQ. The idea is that LinkedIn can't make the data public and freely accessible and then turn around and say that the data isn't public if you're a potential competitor who is using an automated tool to access it in bulk.


that sounds bizarre. why would they order them to do that? what if they re trying to block spammers or sth. What about pages that users want public, but not indexable, e.g. dropbox shared links


Has robots.txt been outlawed now? In what jurisdiction exactly?


I don't believe that robots.txt has ever had the backing of law.


Robots.txt doesn't restrict anything. It's just a request of what should and shouldn't be scraped by search engine spiders.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: