December 8th, 2006
Battle of the bots.
Every bot is supposed to check in the main directory of websites it visits for a file called robots.txt. It’s just a text file that tells bots what files and directories it’s allowed to look at and which ones it isn’t. It’s a simple way for example to keep search bots from looking in your image folder and indexing it. You can tell bots not to look at profile.php which is the file you click on when you view a profile.
Bots like Googlebot and Yahoo!Slurp look at robots.txt and obey it. All what I call “good” bots do. If a bot comes to sas and doesn’t check the robots.txt file it’s automatically a bad bot. Even if its intentions are not malicious it’s at the very least poorly programmed and isn’t following the rules.
Leave a Reply
You must be logged in to post a comment.