Kort om innstillinger i robots.txt

Allow full access:

User-agent: * Disallow:

Block all access:

User-agent: * Disallow: /

Block folder:

User-agent: *
Disallow: /folder/

Block one file
User-agent: *
Disallow: /file.html

Override for all sites on server: