# Tells Scanning Robots Where They Are And Are Not Welcome # User-agent:can also specify by name; "*" is for everyone # Disallow: disallow if this matches first part of requested path # For now disallow all we can modify this as needed to allow certain crawlers. User-agent: * Allow: /