This has been mentioned a couple of times previously on other sites.

The've been burned by having search accessable documents (with meta data) that can't be found on the White House Web site.

Thus the extensive block list. Now *only* those things that are actually on the web site are accessable (and only through the web site).
David | 12.27.05 - 3:25 pm | #

Ah, that explains it. Thanks.
Alex Eckelberry (Siteowner) | 12.27.05 - 3:48 pm | #

Actually, it doesn't explain it. If a search engine can find something, then surely a human can get to it as well. Search engines only follow the links from each page, how can they do otherwise?
Daniel Patterson | Homepage | 12.28.05 - 9:20 pm | #

"Please note: many files associated with the previous administration have been removed from this server. Some materials may be available through the National Archives and Records Administration website."
Roger | 12.29.05 - 4:00 pm | #

DFD
CUA | Homepage | 01.08.06 - 8:01 am | #