839970 - sitemap extension needs to allow access to data/SiteMapIndex/sitemap* in robots.txt

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Extensions, defect)

Description

[:glob ✱]

generated using google's sitemap test:

Sitemap: https://bugzilla.mozilla.org/page.cgi?id=sitemap/sitemap.xml
Type: Sitemap index
Number of children in this Sitemap index 16
Error details: 0 Errors, 16 Warnings.

Warnings

Url blocked by robots.txt.
Sitemap contains 16 urls which are blocked by robots.txt.

Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap1.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap10.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap11.xml.gz
..

Comment 1

[David Lawrence [:dkl]]

Added 

Allow: /*data/SiteMapIndex/sitemap*.xml.gz

Committing to: bzr+ssh://[e-mail]/bmo/4.0             
modified robots.txt
Committed revision 8487. 

Committing to: bzr+ssh://[e-mail]/bmo/4.2             
modified robots.txt
Committed revision 8558.

Comment 2

[:glob ✱]

shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated too?

Comment 3

[David Lawrence [:dkl]]

(In reply to Byron Jones ‹:glob› from comment #2)
> shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated
> too?

Good catch.

Committing to: bzr+ssh://[e-mail]/bmo/4.0         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8488.

Committing to: bzr+ssh://[e-mail]/bmo/4         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8562.