Webmaster level: All Do you know how Google's crawler, Googlebot, handles conflicting directives in your robots.txt file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot's favorite song? The answers to these questions )except for the last one :((, along with lots of other information about controlling the crawling and indexing of your site, are now available on code.google.