Multilingual servers support

Some web-servers can handle language negotiation for documents language. In this case, for one URL exist several copies in different languages.

For indexing all pages of such servers, VaryLang command is used. It specifies a space-separated list of languages. These languages will be used for indexing URLs with multi-language versions.

Usage example:

VaryLang "ru en fr"

index will fetch all document copies in Russian, English and French languages.