robots.txt

[rob]

Do I need to put any code on my index page in addition to storing the robots.txt page in the right place on my server?

All the sites I read just tell me to have the robots.txt placed in the root file.

Is that really all you do?

[James]

I don't think you need a robots.txt at all, unless you want to exclude search engines or impose certain restrictions.

Provided your site is in the search engines' index, and in the absence of the robots.txt files, the default position appears to be "index all, follow all".

Just my opinion though.

[David]

I agree with James. You only need robots.txt if you want to "hide" part of your site from robots. If you want all of your site indexed, just leave it out.

[rob]

So my error logs keep showing that whatever is trying to crawl my site cannot find robots.txt

[Fri Mar 28 18:31:07 2008] [error] [client 66.249.72.141] File does not exist: /home/.../robots.txt

I must get at least 20 of these a day.

Do you think this error is stopping some of the spiders from crawling the site?

[David]

Do you think this error is stopping some of the spiders from crawling the site?

No, spiders look for robots.txt (make a default request for it) when they first enter the site. If they don't find it, they crawl the site anyway. The only real purpose for robots.txt is to tell spiders not to crawl parts of the site. You can stop the errors just by having an empty robots.txt file if you want.