How to do keyword research Learn SEO Tips & Tricks

Robots Txt File
Creating Robots file (written in notepad with name robots.txt and added through ftp filezilla.)
User-agent: *
Disallow: /images/
Disallow: /folder/
Disallow: /images/
Robots file is added in our site for blocking (hiding) some of our pages of the site. Then search engines will not able to crawl those pages. Robots file is used for hiding some of your bad pages of the site from search engine crawlings, because it may drop our rankings, so we hide those pages from search engines.
Example:
User-agent: * (all search engines)
Disallow: /seo_courses.html
Disallow: /images/
User-agent: googlebot (for blocking google)
Disallow: /images/
User agent: yahooseeker (for blocking yahoo)
Disallow: /images/
User agent: msnbot (for blocking bing)
Disallow: /folder/
Read More Here : http://en.wikipedia.org /wiki/Robots_exclusion_ standard