Just a note to say that, as the link above explains, the robots.txt file will only exclude 'bots which (or should I say 'who') care to obey the rules. As far as I know, all the 'major' search engines do obey, but your installation will still get 'crawled' by those who don't . . .