It's a courtesy, afaict. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		kohtatsu on Sept 1, 2020 \| parent \| context \| favorite \| on: Applebot It's a courtesy, afaict.

moonchild on Sept 1, 2020 [–]

Rather the opposite. Many robots.txt will disallow * but allow googlebot. This allows them to crawl those sites without getting flak for disregarding robots.txt completely.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact