Currently, spiders are being asked to totally ignore MDC. This isn't exactly ideal. Instead, the robots.txt that MindTouch provided should be used. Pete should be sending that along shortly.
User-Agent: * Crawl-delay: 5 Sitemap: sitemap.xml Request-rate: 1/5 Disallow: /@api/deki/* Disallow: /*feed=rss Disallow: /*type=feed Disallow: /skins Disallow: /template:
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.