Last Comment Bug 839970 - sitemap extension needs to allow access to data/SiteMapIndex/sitemap* in robots.txt
: sitemap extension needs to allow access to data/SiteMapIndex/sitemap* in robo...
Status: RESOLVED FIXED
:
Product: bugzilla.mozilla.org
Classification: Other
Component: Extensions: Other (show other bugs)
: Production
: x86 Mac OS X
: -- normal (vote)
: ---
Assigned To: David Lawrence [:dkl]
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-02-11 00:28 PST by Byron Jones ‹:glob› [PTO until 2016-10-10]
Modified: 2013-02-15 08:31 PST (History)
3 users (show)
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description Byron Jones ‹:glob› [PTO until 2016-10-10] 2013-02-11 00:28:49 PST
generated using google's sitemap test:

Sitemap: https://bugzilla.mozilla.org/page.cgi?id=sitemap/sitemap.xml
Type: Sitemap index
Number of children in this Sitemap index 16
Error details: 0 Errors, 16 Warnings.

Warnings

Url blocked by robots.txt.
Sitemap contains 16 urls which are blocked by robots.txt.

Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap1.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap10.xml.gz
Value: https://bugzilla.mozilla.org/data/SiteMapIndex/sitemap11.xml.gz
..
Comment 1 David Lawrence [:dkl] 2013-02-14 15:28:43 PST
Added 

Allow: /*data/SiteMapIndex/sitemap*.xml.gz

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.0             
modified robots.txt
Committed revision 8487. 

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.2             
modified robots.txt
Committed revision 8558.
Comment 2 Byron Jones ‹:glob› [PTO until 2016-10-10] 2013-02-14 19:21:14 PST
shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated too?
Comment 3 David Lawrence [:dkl] 2013-02-15 08:31:25 PST
(In reply to Byron Jones ‹:glob› from comment #2)
> shouldn't extensions/SiteMapIndex/Extension.pm:_fix_robots_txt() be updated
> too?

Good catch.

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4.0         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8488.

Committing to: bzr+ssh://dlawrence%40mozilla.com@bzr.mozilla.org/bmo/4         
modified extensions/SiteMapIndex/robots.txt
Committed revision 8562.

Note You need to log in before you can comment on or make changes to this bug.