Dev and stage websites show up in Google search results

RESOLVED FIXED

Status

Websites
webifyme.org
RESOLVED FIXED
7 years ago
7 years ago

People

(Reporter: rik, Assigned: oremj)

Tracking

Details

(Whiteboard: [seo])

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
1) Go to http://www.google.com/search?q=webifyme
2) I see http://webifyme-dev.allizom.org as the 7th result

We should add a robots.txt to prevent indexing of non-production websites.

Comment 1

7 years ago
(In reply to comment #0)
> 1) Go to http://www.google.com/search?q=webifyme
> 2) I see http://webifyme-dev.allizom.org as the 7th result
> 
> We should add a robots.txt to prevent indexing of non-production websites.

+1. It would be bad if this site still shows up after we launch, worse if it outranks the production site because the staging site has been around longer. I'm not sure how high a risk this is, but I still think this is the best practice here. 

Further, this should be standard protocol on all staging sites. 

Chris - can you recommend next steps here or route this to the appropriate dev?
Assignee: nobody → chrismore.bugzilla

Comment 2

7 years ago
Filed IT bug 673530 to have a robots.txt file added to webifyme dev and stage websites.

Updated

7 years ago
Assignee: chrismore.bugzilla → jeremy.orem+bugs
(Assignee)

Comment 3

7 years ago
Addons has a good way of doing this:


https://github.com/jbalogh/zamboni/blob/master/apps/amo/views.py#L190
https://github.com/jbalogh/zamboni/blob/master/apps/amo/urls.py#L21
https://github.com/jbalogh/zamboni/blob/master/settings.py#L895
Assignee: jeremy.orem+bugs → nobody
(Assignee)

Updated

7 years ago
Assignee: nobody → chrismore.bugzilla
(Assignee)

Updated

7 years ago
Duplicate of this bug: 673530
(Reporter)

Comment 5

7 years ago
We should backport this into playdoh.
(In reply to comment #5)
> We should backport this into playdoh.

Great minds think alike... I thought the same thing when I read Jeremy's comment. Rik, will you file an Issue on playdoh's github? Merci!
(Reporter)

Comment 7

7 years ago
Filed https://github.com/mozilla/playdoh/issues/53.

Comment 8

7 years ago
is a robots.txt going to be added to dev/stage in the meantime?

Updated

7 years ago
Assignee: chrismore.bugzilla → jeremy.orem+bugs
(Assignee)

Comment 9

7 years ago
Created attachment 549153 [details] [diff] [review]
Adds simple robots.txt support

Sure, we just need to apply this patch. Should I push it to master?
Comment on attachment 549153 [details] [diff] [review]
Adds simple robots.txt support

Yes.
Attachment #549153 - Flags: review+

Comment 11

7 years ago
Also a reminder to remove that file once we go live. I'm not sure if a separate bug should be filed for that or if it can be combined into this process somehow.
(Assignee)

Comment 12

7 years ago
Pushed. That should finish up this bug. Looks like adding this to playdoh will be addressed elsewhere.

(In reply to comment #11)
> Also a reminder to remove that file once we go live. I'm not sure if a
> separate bug should be filed for that or if it can be combined into this
> process somehow.

We just need to set ENGAGE_ROBOTS in settings_local.py to True in production.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.