Perhaps Fedbot crawls in a less deterministic manner, uses a lot of different ip... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

wisty on Aug 13, 2011 | parent | context | favorite | on: Fuzzing at scale

Perhaps Fedbot crawls in a less deterministic manner, uses a lot of different ips, and sets user agent to IE?

bigiain on Aug 13, 2011 [–]

I suspect "fedbot" works by calling up google and saying "Hi, it's us again, we've got another white van on the way to the googleplex, have a petabyte or two of the Internet ready for us to collect in 20 minutes. thanks"

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact