User-agent: * Disallow: /admin Allow: / Crawl-Delay: 10 User-agent: Twitterbot Crawl-Delay: 20 User-agent: Yandex Crawl-Delay: 20