Fix robots.txt file on MDC

VERIFIED FIXED

Status

mozilla.org Graveyard
Server Operations
--
critical
VERIFIED FIXED
8 years ago
3 years ago

People

(Reporter: sheppy, Assigned: oremj)

Tracking

Details

(Reporter)

Description

8 years ago
Currently, spiders are being asked to totally ignore MDC. This isn't exactly ideal. Instead, the robots.txt that MindTouch provided should be used. Pete should be sending that along shortly.
(Reporter)

Comment 1

8 years ago
User-Agent: *
Crawl-delay: 5
Sitemap: sitemap.xml
Request-rate: 1/5

Disallow: /@api/deki/*
Disallow: /*feed=rss
Disallow: /*type=feed
Disallow: /skins
Disallow: /template:
(Assignee)

Comment 2

8 years ago
Fixed.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.