Given that FB's bot identifies itself, no, eventually some websites will present... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		wumpus on Aug 24, 2016 \| parent \| context \| favorite \| on: Web Scraping in 2016 Given that FB's bot identifies itself, no, eventually some websites will present og: markup only to FB's bot.

fareesh on Aug 24, 2016 [–]

Couldn't you just spoof the user agent though? Or is there some other mechanism that you can use to verify the bot is Facebook's?

wumpus on Aug 24, 2016 | [–]

IP address.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact