[nemanirc]

I checked my httpd error log in Console and noticed several requests from different IP addresses requesting a "robots.txt" file (they get the error File does not exist: /Library/WebServer/Documents/robots.txt). What is that? Should I be concerned?

[Arkham_c]

Originally posted by nemanirc:
I checked my httpd error log in Console and noticed several requests from different IP addresses requesting a "robots.txt" file (they get the error File does not exist: /Library/WebServer/Documents/robots.txt). What is that? Should I be concerned?

No worries:

http://www.robotstxt.org/wc/robots.html

[bens1901]

Just to add to the excellent post above...

Search engines will look in your root domain for a special file named "robots.txt", which explains how spiders (auto-surfing programs) should treat your site. You can tell them to not index a particular directory, ignore a particular file, index the entire site, or basically anything.