820850 - [mozillians] Change robots.txt on production.

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Giorgos Logiotatidis [:giorgos]]

Hello IT,

Please change /robots.txt on mozillians.org production so that it points our repository's robots.txt

Comment 1

[Chris Turra [:cturra]]

:giorgos - this has actually been in place for some time now.

  RewriteRule ^/robots.txt$ /data/www/mozillians.org/mozillians/media/robots.txt

Comment 2

[Giorgos Logiotatidis [:giorgos]]

My bad, thanks for pointing that out.

Comment 3

[Giorgos Logiotatidis [:giorgos]]

Is this the same for dev and stage?

If yes this is not the expected behavior as we want different robots.txt file for prod and different for dev and stage. Can you make prod to point to robots-prod.txt please?

Thanks!

Comment 4

[Chris Turra [:cturra]]

:giorgos - where can i find robots-prod.txt?

[[e-mail]1 mozillians.org]# find mozillians -iname robots-prod.txt  | wc -l
0

Comment 5

[Giorgos Logiotatidis [:giorgos]]

:cturra 

It's now in /media/

Thanks!

Comment 6

[Chris Turra [:cturra]]

:giorgos - i have committed a change for dev and stage to serve the new robots-prod.txt. note: you will need to do a chief push for this new robots-prod.txt file to be available in stage.

once you have tested everything looks as expected, let me know and i will do a push to prod for this change.

Comment 7

[Giorgos Logiotatidis [:giorgos]]

OK, I messed up the communications at some point, so let's try again:

 * Dev and Stage should point to /media/robots.txt
 * Prod should point to /media/robots-prod.txt

Thanks and sorry for the trouble.

Comment 8

[Chris Turra [:cturra]]

:giorgos - thnx for the clarification, it's entirely possible it was my overlooking versus your communication :)

i have updated the rewrite rules to point dev/stage at /media/robots.txt and prod to /media/robots-prod.txt per your request above. this should all be ready for testing and looks good at first glance on my end.

$ curl https://mozillians.org/robots.txt
User-agent: *
Disallow:

$ curl https://mozillians.allizom.org/robots.txt
User-agent: *
Disallow: /

$ curl https://mozillians-dev.allizom.org/robots.txt
User-agent: *
Disallow: /

Comment 9

[Giorgos Logiotatidis [:giorgos]]

Verified, Thanks!