The default bug view has changed. See this FAQ.

sitemap extension needs to allow access to data/SiteMapIndex/sitemap* in robots.txt

RESOLVED FIXED

Status

()

bugzilla.mozilla.org
Extensions: Other
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: glob, Assigned: dkl)

Tracking

Production
x86
Mac OS X

Details

(Reporter)

Description

4 years ago
generated using google's sitemap test:

Sitemap: https://bugzilla.mozilla.org/page.cgi?id=sitemap/sitemap.xml
Type: Sitemap index
Number of children in this Sitemap index 16
Error details: 0 Errors, 16 Warnings.

Warnings

Url blocked by robots.txt.
Sitemap contains 16 urls which are blocked by robots.txt.

Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap1.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap10.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap11.xml.gz
..
(Assignee)

Updated

4 years ago
Assignee: nobody → dkl
Status: NEW → ASSIGNED
(Assignee)

Comment 1

4 years ago
Added 

Allow: /*data/SiteMapIndex/sitemap*.xml.gz

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.0             
modified robots.txt
Committed revision 8487. 

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.2             
modified robots.txt
Committed revision 8558.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 2

4 years ago
shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated too?
(Assignee)

Comment 3

4 years ago
(In reply to Byron Jones ‹:glob› from comment #2)
> shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated
> too?

Good catch.

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.0         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8488.

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8562.
You need to log in before you can comment on or make changes to this bug.