A sitemap is a must-have “accessory” for any website. If your site is published on WordPress you can install Google’s sitemap.xml plugin. If you use other software there are plugins and add-ons to do the same.

These plugins will automatically rebuild your sitemap every time a new page is created and PING all major search engine. This is a great SEO add on. You can also visit Google’s Webmaster Tools, register for a free account and submit your sitemap right there and then.

Robots.txt

The Robots.txt file is a text-based file on your site. It tells search engine robots which pages you don’t want them to visit. Having said this, for SEO purposes the Robots.txt file is a must-have. It is also important to note that this text file is in no way a means to protect your site against crawlers and hackers.

If you site features sensitive data, and protected download pages you need to rely on tougher security measures than the simple Robots.txt file.

Position of Robots.txt

Your files must be hosted in the main directory for search engines to be able to crawl it. Your main directory is your http://mydomain.com/robots.txt location. If the bots don’t find the file there, they simply assume it isn’t existent on your site, and end up indexing everything they find, even the stuff they shouldn’t.

Robots.txt and SEO

For SEO purposes the Robots.txt file can have an impact on how your site is crawled and indexed, therefore making it another essential tool to have in your arsenal for search engine domination. Jump onto Google’s Webmaster Tools to create your robots.txt file.