Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

We develop and host a bunch of extranets, which without login consist of your typical authentication page. We put a robots.txt file there, and the only sites that link there are our customers companies home sites.

Google still indexes them. The definition of "relevant" here defies my wildest imagination.



robots.txt is not about indexing. It's about crawling.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: