Kort om innstillinger i robots.txt
Allow full access:
User-agent: * Disallow:
Block all access:
User-agent: * Disallow: /
Block folder:
User-agent: *
Disallow: /folder/
Block one file
User-agent: *
Disallow: /file.html
Override for all sites on server:
- Lag en "global' robots.txt/robort-override.txt i en katalog tilgjengelig for apache2, f. eks i var/www.
- Legg til alias som overstyrer i apache2.conf:
Alias /robots.txt /var/www/robots.txt