Closed Bug 839970 Opened 11 years ago Closed 11 years ago

sitemap extension needs to allow access to data/SiteMapIndex/sitemap* in robots.txt

Categories

(bugzilla.mozilla.org :: Extensions, defect)

Product:
bugzilla.mozilla.org
Component:
Extensions
Version:
Production
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: glob, Assigned: dkl)

Details

generated using google's sitemap test:

Sitemap: https://bugzilla.mozilla.org/page.cgi?id=sitemap/sitemap.xml
Type: Sitemap index
Number of children in this Sitemap index 16
Error details: 0 Errors, 16 Warnings.

Warnings

Url blocked by robots.txt.
Sitemap contains 16 urls which are blocked by robots.txt.

Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap1.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap10.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap11.xml.gz
..
Assignee: nobody → dkl
Status: NEW → ASSIGNED
Added 

Allow: /*data/SiteMapIndex/sitemap*.xml.gz

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.0             
modified robots.txt
Committed revision 8487. 

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.2             
modified robots.txt
Committed revision 8558.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated too?
(In reply to Byron Jones ‹:glob› from comment #2)
> shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated
> too?

Good catch.

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.0         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8488.

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8562.
Component: Extensions: Other → Extensions
You need to log in before you can comment on or make changes to this bug.