About robots.txt

Web site owners use the robots.txt file to give instructions about their site to web robots robotstxt.org

By default CrawlForMe follows the robots.txt instructions of the crawled website but you can also choice to ignore it and analyse all the pages of your website.

Example of robots.txt

Here is the content of the robots.txt of this website :

CrawlForMe

Here is a link ignored by the robots.txt : Link ignored by robots.txt.
You will find this link in the ignored tab of the demo report with the description : "ignored by robots.txt".

Useful links

More informations about robots.txt :

Robots generator

Some website help you to generate robots.txt file :

About ignored pattern

An another way to ignore links during your crawling is to use the ignored pattern functionality of CrawlForMe. Select the links that you want to ignore (by using pattern) and optimize your crawling report and your budget.

More informations about pattern

Here are links ignored by the pattern ("/ignored"):

You will find these links in the ignored tab of the demo report.