Robots.txt, An Online Marketers Friend Or Foe? Free PLR Article robots.txt is possibly the most miss understood file that a website can contain. Many people think that by using a robots.txt file on their website they are protecting pages and used by search engine spiders to determine which parts of your website they should index and which folders/pages that you, the website owner, dont want listed in search engine indexes. Why would not want pages indexed? There are many reasons why training pages and such like. If you are an Internet Marketer selling your own ebook or other digital product, you wouldnt want your thank you pages indexed either! And this is where the misunderstanding comes to the fore, and robots.txt ...