$robots txt

Build a robots.txt from a few simple rules. Each group sets a User-agent plus its Allow/Disallow paths. Optional Crawl-delay and Sitemap URL.