User-agent: * Disallow: /public/ User-agent: Yandex Crawl-delay: 10 # specifies a 10 second timeout