I wish every company which crawls had such a page. Maybe a centralized directory... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		qwertox on Aug 7, 2023 \| parent \| context \| favorite \| on: GPTBot – OpenAI’s Web Crawler I wish every company which crawls had such a page. Maybe a centralized directory of crawlers would be a good for the internet.

Gasp0de on Aug 7, 2023 [–]

I mean, technically you can just disallow all robots in your robots.txt and only allow those of which you are aware and you like?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact