744030 - [SEO][Bedrock] Google retrieves descriptions from dmoz; needs X-Robots-Tag: noodp

Status: VERIFIED FIXED

(www.mozilla.org :: General, defect)

Description

[Kohei Yoshino [:kohei]]

+++ This bug was initially created as a clone of Bug #682023 +++

I've noticed that Google cites the snippet of the Firefox home page from ODP/dmoz.
https://www.google.com/search?q=Firefox&hl=en

The Bedrock site needs
> X-Robots-Tag: noodp

Comment 2

[Anthony Ricaud (:rik)]

Taking this to work on it next week.

Comment 3

[Anthony Ricaud (:rik)]

I have an idea to fix that properly and I need to discuss with someone else.

Comment 4

[Anthony Ricaud (:rik)]

FYI, there is a pull request here: https://github.com/jsocol/commonware/pull/12

Comment 5

[Mike Alexis [:malexis]]

(In reply to Anthony Ricaud (:rik) from comment #4)
> FYI, there is a pull request here:
> https://github.com/jsocol/commonware/pull/12

Were these changes ever pushed to prod? Can we close this bug?

Comment 6

[Anthony Ricaud (:rik)]

Nope, not pushed. I have to act on the review comments.

Comment 7

[James Socol [:jsocol, :james]]

Rik: I'm stealing this to polish up the pull req and get it merged. Will reassign back to you when I push a new version of commonware to PyPI.

Comment 8

[Anthony Ricaud (:rik)]

Looks like you released 0.4.0. As it's not critical, I'm gonna mark this as a mentored bug.

What is involved here?
- Update the commonware lib to >= 0.4.0
- Install commonware.response.middleware.RobotsTagHeader
- Change X_ROBOTS_DEFAULT

Comment 9

[u465284]

requirements/prod.txt contains this line:

-e git://github.com/jsocol/commonware.git#egg=commonware

So we are already using last version of commonware, right?

Comment 10

[u465284]

Ok, got it: we are using a sub-submodule (a submobule of playdoh-lib, which in turn is a submodule of bedrock). So to get the last commonware version I should make a pull request to playdoh-lib, wait it gets merged, and then make a pull request to bedrock.

Rik: can you help me with this? Anyone else?

Comment 11

[Kohei Yoshino [:kohei]]

If it's difficult to modify HTTP headers, you can add the following meta tag to each page instead, like Bug 848839:

<meta name="robots" content="noodp">

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

Comment 12

[u465284]

This should be much simpler, and of course doesnt' require a commonware upgrade. Any opinions?

Comment 13

[James Socol [:jsocol, :james]]

(In reply to Giuseppe from comment #12)
> This should be much simpler, and of course doesnt' require a commonware
> upgrade. Any opinions?

The "right" fix is to upgrade commonware (we need to keep playdoh-lib pretty up-to-date). It might not be the fastest fix, though.

Comment 14

[u465284]

(In reply to James Socol [:jsocol, :james] from comment #13)
> The "right" fix is to upgrade commonware (we need to keep playdoh-lib pretty
> up-to-date). It might not be the fastest fix, though.

I can make a pull request, but we are blocked until it gets accepted.

Comment 15

[u465284]

Here is the PR: https://github.com/mozilla/playdoh-lib/pull/35

Comment 16

[Kohei Yoshino [:kohei]]

This is already fixed on master.

Comment 17

[Kohei Yoshino [:kohei]]

Nope, it's still an issue on Bedrock.

Comment 18

[Kohei Yoshino [:kohei]]

What's the status here? I still cannot see the X-Robots-Tag header. Should we fix the issue on Bedrock?

Comment 19

[Chris More [:cmore]]

Josh: can you check this out and see what you think?

Comment 20

[Josh Mize [:jgmize]]

I verified that bedrock's version of playdoh-lib has commonware 0.4.2, which has a RobotsTagHeader middleware that sets the "X-Robots-Tag: noodp" header by default, but this middleware is not currently enabled in bedrock. I've submitted a PR to enable it.

Comment 21

[Josh Mize [:jgmize]]

Attachment: PR to enable RobotsTagHeader middleware

Comment 22

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/af4bd97afec835c560099b169a151fac767af891
Fix Bug 744030 - Enable RobotsTagHeader middleware

https://github.com/mozilla/bedrock/commit/2a3dfde1a36c66c7d206e897a2d60933233b795b
Merge pull request #1351 from jgmize/bug-744030-x-robots-tag-noodp

Fix Bug 744030 - Enable RobotsTagHeader middleware

Comment 23

[Kohei Yoshino [:kohei]]

$ curl -I https://www-dev.allizom.org/en-US/
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: bedrock1.dev.webapp.phx1.mozilla.com
Vary: Accept-Encoding
Cache-Control: max-age=600
Content-Type: text/html; charset=utf-8
Date: Tue, 29 Oct 2013 14:05:37 GMT
Expires: Tue, 29 Oct 2013 14:15:39 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
X-Robots-Tag: noodp
X-Frame-Options: DENY

:)

Comment 24

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED on trunk:

https://www-dev.allizom.org/en-US/

Comment 25

[Stephen Donner [:stephend] Not actively reading bugmail]

Attachment: Post-fix screenshot