A robots.txt file tells search engine robots what they may and may not access from your site. A robots.txt file does not enforces access to your site. It is merely a suggestion to search engine robots that respect these robots.txt rules. These search engine robots, also called bots, that crawl the web are fully automated. Before they access pages of your site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For these bad boys you can setup a bot trap that is making use of robots.txt. Anything that needs to be confidential is best to password protected.