I bet NSA didn't even have to make any network requests on this one. They either got backdoors in most cache servers or have already duplicated their contents.
I would be shocked if the NSA hasn't been running their own crawlers for years. Because, you know, terrorists may be chatting on pages not allowed by robots.txt. And just because other crawlers respect it, doesn't mean we need to...
