GEN:SID 1:1857
Message WEB-MISC robot.txt access
Summary This event is generated when a client is requesting the file "robot.txt"
from a web server.
Impact Information Disclosure. This file may contain data that could provide an
attacker with information that could assist in an attack on the server.
Detailed Information In the early days of the web, when search engines first began indexing
sites, it was often desirable to tell the indexing programs, referred
to as robots, not to index certain parts of a site. A standarized
method of accomplishing this was created; by placing a file called
"robot.txt" or "robots.txt" in the root of your web site which search
engines could read and which would tell them what parts of your site you
did not want indexed. However, this file can also be very valuable to
potential attackers if it contains information such as restricted
directories, cgi-bin locations, etc.
Affected Systems Any web site that uses this method to communicate with robots.
Attack Scenarios An attacker can read the "robot.txt" file and use any sensitive data in
it to profile your site in preparation for an attack.
Ease of Attack Simple. No exploit software required. Any browser can request a copy of
"robot.txt" from the server.
Corrective Action Ensure that your "robot.txt" file, if you need one, does not contain any
sensitive data.
Additional References  
Rule References nessus: 10302