Robots
Name
Robots -- allows using robots.txt
indexer.conf
Synopsis
Robots {yes | no}
Description
Allows/disallows using robots.txt and <META NAME="robots">
exclusions. Use "no", for example for link validation of your server(s).
Command may be used several times before "Server" command and
takes effect till the end of config file or till next Robots command.
Default value is "yes".