Robots.txt
Posted by David | Filed under Web Design Tips
A Robots.txt is especially important for those of us with WordPress or similar CMS that have directories we don’t really want Google poking its spider nose into! For example I don’t want Google rummaging through wp-admin/ wp-content/ or wp-includes/ so how can I stop that happening?
Simple – Googlebots along with many other good bots obey a file called the Robots.txt which you should upload to the root of your web directory. The easiest way to write a robots.txt is to do so in NotePad and save it as “Robot.txt” then upload to the root of your server.
In the file you should include the following to ban all robots from entering the directory
User-agent: *
Disallow: /thisdirectory
User-agent: *
Disallow: /thatdirectory
For WordPress this would be:
User-agent: *
Disallow: /wp-content
User-agent: *
Disallow: /wp-admin
User-agent: *
Disallow: /wp-includes
You can also ban the robots from looking at specific files as follows:
User-agent: *
Disallow: /thatfile.html
You can also band specific spider e.g. the Google Images Spiders or just ban a Googlebot.
User-agent: Googlebot-Images
Disallow: /mysecretimages
User-agent: Googlebot
Disallow: /nogoogle
- Search Engine Optimization Tips Recently a couple of my clients have asked me about getting their websites to the top of the search engine listings, and I’ve had to...
- The “Do Not’s” of SEOing How not to SEO! Following my article on how to SEO I thought I should write a second one telling you how not to! Firstly...
Tags: Googlebots, Robots, Robots.txt, Search Spiders, Spiders