User-agent: GoogleBot Disallow: User-agent: MSNBot Disallow: User-agent: Slurp Disallow: User-agent: IconSurf Disallow: / User-agent: BecomeBot Disallow: / # OmniExplorer BOT uses a bad way (non compliant) of disallowing # the crawl. User-Agent: OmniExplorer_Bot/1.09 Disallow: * User-agent: * Disallow: /