It's public *right now*. It won't be once Yahoo pulls the plug on Delicious. | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

sabat on Dec 16, 2010 | parent | context | favorite | on: Yahoo is Shutting Down Del.icio.us

It's public right now. It won't be once Yahoo pulls the plug on Delicious.

Tichy on Dec 16, 2010 [–]

User-agent: * Disallow: /

I don't remember the robots.txt rules for sure, but doesn't that mean they don't allow crawlers at all?

ivank on Dec 16, 2010 | | [–]

That's the rule for crawlers that aren't Slurp, Googlebot, Teoma, or msnbot.

Tichy on Dec 16, 2010 | | | [–]

I noticed the extra rules, but I am neither Slurp, Googlebot, Teoma nor msnbot :-(

mfukar on Dec 17, 2010 | | [–]

robots.txt is merely a suggestion.

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact