767708 - QA and deploy BrowserID train-2012.06.22 to production

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Lloyd Hilaiel [:lloyd]]

Version: 5d3eeb7642 (0.2012.06.22.1) branch train-2012.06.22

Tests pass: http://travis-ci.org/#!/mozilla/browserid/builds/1572639

ChangeLog including issues resolved: 
https://github.com/mozilla/browserid/blob/train-2012.06.22/ChangeLog#L1-18

[QA] Suggested additional areas of focus for QA:
  * more focus on 123done.org, all login/logout scenarios, IE8/9 focus useful as well.

[ops] deployment issues:
  * must be deployed with persona URLs in beta and production!  see https://github.com/mozilla/browserid/issues/1741
  * __heartbeat__ urls now support "deep" heartbeats, requested by ops for router process: https://github.com/mozilla/browserid/issues/1767

Comment 1

[Lloyd Hilaiel [:lloyd]]

sorry, relevant tests are here: http://travis-ci.org/#!/mozilla/browserid/builds/1680043

Comment 2

[James Bonacci [:jbonacci]]

:lloyd it would be help to QA and SV to have the complete list of all new URLs in Dev and Stage, and their mapping to current sites for backward compatibility testing.
Thanks

Comment 3

[Lloyd Hilaiel [:lloyd]]

:jbonacci, what specifically do you need above and beyond https://github.com/mozilla/browserid/issues/1741

Comment 4

[James Bonacci [:jbonacci]]

Thanks :lloyd, I missed this line from above:
"[ops] deployment issues:
  * must be deployed with persona URLs in beta and production!  see https://github.com/mozilla/browserid/issues/1741"


I wanted something to show the team from SV for testing purposes.

Comment 5

[Lloyd Hilaiel [:lloyd]]

cool.  :jbonacci, to augment that, here are the intended visible behaviors of various urls:

1. http(s)?://anosrep.org should HTTP redirect to http(s)?://login.anosrep.org
2. http(s)?://diresworb.org/verify SHOULD NOT REDIRECT
3. all other urls on http(s)?://diresworb.org should HTTP redirect to http(s)?://login.anosrep.org
4. all static resources should serve from https://static.login.anosrep.org

for production, simply s/anosrep/persona/

Does this give you all the context you need?

Comment 6

[Pete Fritchman [:petef]]

Zeus changes for production:

* add {scl2,phx1}-persona vServer (according to wiki)
* add persona-redirect TS rule to existing *-idweb vServer
* update route-browserid-verifier TS to include hostname routing

Comment 7

[Pete Fritchman [:petef]]

Other TODO:

Prod dynect: change health checks for browserid.org VIPs to expect a 302

Comment 8

[Pete Fritchman [:petef]]

(In reply to Pete Fritchman [:petef] from comment #7)
> Other TODO:
> 
> Prod dynect: change health checks for browserid.org VIPs to expect a 302

Nevermind -- we updated persona-redirect to exclude /__heartbeat__ paths.

Comment 9

[Pete Fritchman [:petef]]

Prod todo: adjust watchmouse monitoring

Comment 10

[Gene Wood [:gene]]

I've updated staging with 5d3eeb7642

Comment 11

[James Bonacci [:jbonacci]]

A quick check shows that both http://diresworb.org and https://diresworb.org redirect to https://login.anosrep.org/
Nice!
And version: https://login.anosrep.org/ver.txt
says 
5d3eeb7 remove IE8 submit on enter fix from changelog, it was backed out
locale svn r106791

Services QE accepts this train for testing in Stage...

Comment 12

[James Bonacci [:jbonacci]]

SV team:

Services QA Release Wiki is here:
https://wiki.mozilla.org/QA/Browser_Technologies/Services/Releases/BrowserID_06222012

The test spreadsheet is here:
https://docs.google.com/spreadsheet/ccc?key=0As8GUPwNNWMFdEstLXkzbE5lZEM5SV9IRHgzUExBU3c#gid=0

Comment 13

[John Morrison [:jrgm]]

(In reply to Lloyd Hilaiel [:lloyd] from comment #5)
> cool.  :jbonacci, to augment that, here are the intended visible behaviors
> of various urls:

1. http(s)?://anosrep.org should HTTP redirect to http(s)?://login.anosrep.org

- http://anosrep.org returns 301 to https://anosrep.org
- https://anosrep.org returns 200 OK (i.e., NOT REDIRECTED)

2. http(s)?://diresworb.org/verify SHOULD NOT REDIRECT

- GET http://diresworb.org/verify returns 301 to https://diresworb.org/verify
  (This actually seems correct to me, and is currently what 
  http://browserid.org does).

- POST http://diresworb.org/verify returns 400 Bad Non-SSL Requests Please Use
  HTTPS, which is a zeus rule (can't find the bug number at the moment where
  we put that in).

- https://diresworb.org/verify returns 404 to GET (which is *correct* for
  GET), and handles POST without a redirect (again correct)

3. all other urls on http(s)?://diresworb.org should HTTP redirect to http(s)?://login.anosrep.org

- http://diresworb.org/wsapi/session_context returns 301 to
  https://diresworb.org/wsapi/session_context

- https://diresworb.org/wsapi/session_context returns 302 to
  https://login.anosrep.org/wsapi/session_context

- https://login.anosrep.org/wsapi/session_context returns 200 OK with a
  correct context payload

4. all static resources should serve from https://static.login.anosrep.org

- I'm not seeing any resources loaded from http(s)?://static.login.anosrep.org

----------------------------------------------------------------------

So, 2 and 3 seem okay, but 1 and 4 are not correct per comment #5

Comment 14

[Pete Fritchman [:petef]]

I think the "public_url" setting in production.json needs to be flipped from diresworb.org to login.anosrep.org.

Comment 15

[John Morrison [:jrgm]]

(In reply to Pete Fritchman [:petef] from comment #14)
> I think the "public_url" setting in production.json needs to be flipped from
> diresworb.org to login.anosrep.org.

Probably 

https://github.com/mozilla/browserid/issues/1845
https://github.com/mozilla/browserid/issues/1846
https://github.com/mozilla/browserid/issues/1847

all go away with this change. In the meantime, I've been testing with an awsbox on train-2012.06.22.

Comment 16

[Gene Wood [:gene]]

I've updated public_url in sysadmins r41914 in response to Comment 14. I've restarted the systems.

Up next, updating l10n-preview and client[4-9].scl2.svc.mozilla.com

Comment 17

[Gene Wood [:gene]]

l10n-preview has been updated, still working on clientN.scl2.svc.mozilla.com

Comment 18

[James Bonacci [:jbonacci]]

Quickly verified the l10n-preview environment:
5d3eeb7 remove IE8 submit on enter fix from changelog, it was backed out
locale svn r106818

Thanks.

Comment 19

[Gene Wood [:gene]]

clientN.scl2.svc.mozilla.com machines have been updated

Comment 20

[Lloyd Hilaiel [:lloyd]]

It sounds like the following are required: 

1. anosrep.org needs to redirect to login.anosrep.org
2. in order to get static resources serving from another domain a new configuration variable is required on the browserid process - https://github.com/mozilla/browserid/blob/dev/lib/configuration.js#L65-68

#2 is not a hard requirement for this train, but it's why we're not seeing static resources served from static.login.anosrep.org in the staging environment (discovered above by jrgm).

Comment 21

[Gene Wood [:gene]]

There was a bug in my update method that missed client4 and client9. I've no done client4 and client9 which covers all of them

[gene@boris ~]$  for i in `seq 4 9`; do echo client${i}.scl2.svc.mozilla.com; ssh root@client${i}.scl2.svc.mozilla.com 'rpm -q browserid-server';doneclient4.scl2.svc.mozilla.com
browserid-server-0.2012.06.22-1.el6_106791.x86_64
client5.scl2.svc.mozilla.com
browserid-server-0.2012.06.22-1.el6_106791.x86_64
client6.scl2.svc.mozilla.com
browserid-server-0.2012.06.22-1.el6_106791.x86_64
client7.scl2.svc.mozilla.com
browserid-server-0.2012.06.22-1.el6_106791.x86_64
client8.scl2.svc.mozilla.com
browserid-server-0.2012.06.22-1.el6_106791.x86_64
client9.scl2.svc.mozilla.com
browserid-server-0.2012.06.22-1.el6_106791.x86_64

Comment 22

[Gene Wood [:gene]]

Regarding Comment 20 :

1. Done
modified the Zeus Rule "persona-redirect" which is used by the "stage-idweb" Virtual Server. The modification causes the redirect to not occur if the original host header is login.anosrep.org

modified the rule list associated with "stage-anosrep" by adding the rule "persona-redirect" to the rule list

2. Not yet done

Comment 23

[Gene Wood [:gene]]

2. Done
Committed new variable for stage in sysadmins r41979
Ran puppet across all stage bid systems
Restarted all processes

Comment 24

[Gene Wood [:gene]]

Is there a static.login.anosrep.org URL that I can test with to see if it's working?

Comment 25

[Gene Wood [:gene]]

I've enabled Multicast in zeus in stage now that netops has completed bug 766561 

Traffic IP Group: stage-anosrep (stage) 
* https://zlb1.pub.scl2.stage.svc.mozilla.com:9090/apps/zxtm/index.fcgi?name=stage-anosrep&section=Traffic%20IP%20Groups%3AEdit
* Changed IP Distribution Mode from Single-hosted to Multi-hosted
* Set Multicast IP to : 239.103.1.12
* Set Consider client source port when splitting load? to Yes

Traffic IP Group: stage-anosrep-login (stage) 
* https://zlb1.pub.scl2.stage.svc.mozilla.com:9090/apps/zxtm/index.fcgi?name=stage-anosrep-login&section=Traffic%20IP%20Groups%3AEdit
* Changed IP Distribution Mode from Single-hosted to Multi-hosted
* Set Multicast IP to : 239.103.1.13
* Set Consider client source port when splitting load? to Yes

Comment 26

[Lloyd Hilaiel [:lloyd]]

please redeploy stage with commit b57d41771e2d - train-2012.06.22 - ver. 0.2012.06.22.02

Also please update SVN to merge final translations for this train - at least revision 107047

Note to QA: this update will fix 1905

Comment 27

[James Bonacci [:jbonacci]]

:lloyd and :jrgm
The "hot fix" for #1905 seems to fit the final redeploy for locale support, otherwise,I would not recommend we do it.

And, while we are at it, if we are going with this, what happened to this potential hot fix?
"A hot fix for pancake, for possible inclusion in train-2012.06.22"
https://github.com/mozilla/browserid/pull/1904

Comment 28

[John Morrison [:jrgm]]

(In reply to James Bonacci [:jbonacci] from comment #27)
> 
> And, while we are at it, if we are going with this, what happened to this
> potential hot fix?
> "A hot fix for pancake, for possible inclusion in train-2012.06.22"
> https://github.com/mozilla/browserid/pull/1904

I tried out all the browser/platform I had available and didn't see a problem. We discussed in this mornings triage and agreed to not do this in https://github.com/mozilla/browserid/pull/1904

Comment 29

[Gene Wood [:gene]]

stage, client[4-9] and l10n-preview have been updated with commit b57d41771e2d - train-2012.06.22 - ver. 0.2012.06.22.02

Comment 30

[Gene Wood [:gene]]

Updated production zeus load balancers to prepare for monday's release. Here are the changes I made :


https://zlb1.pub.scl2.svc.mozilla.com:9090/
* Traffic IP Group scl2-persona was setup already and was already configured to use Traffic IP Group : multi-hosted
* Services... Virtual Servers... Create new
** Name : scl2-persona
** Protocol : HTTP
** Port : 443
** Default Pool : scl2-idweb
** Enabled : No
** Listening on : scl2-persona (Traffic IP Group)
** Notes : *.persona.org
*** SSL Decryption
**** Loaded the www.persona.org-EV cert
**** ssl_decrypt : yes
**** default certificate : www.persona.org-EV
*** Rules
**** Created new rule "persona-redirect"
        $host = "login.persona.org";
        $path = http.getPath();
        $hostheader = http.getHostHeader();

        if ($path != "/__heartbeat__") {
           if ($hostheader == "persona.org" ||
               $hostheader == "browserid.org" ||
               $hostheader == "www.browserid.org") {
              $qs = http.getQueryString();
              if ($qs == "") {
                 http.redirect("https://" . $host . $path);
              } else {
                 http.redirect("https://" . $host . $path . "?" . $qs);
              }
           }
        }
**** Modified rule "route-verifier-requests" from
        $path = http.getPath();

        if ($path == "/verify") {
          $cur_pool = connection.getVirtualServer();
          $new_pool = string.replace($cur_pool, "-idweb", "-idverifier");
          pool.use($new_pool);
        }
to
        $path = http.getPath();
        $host = http.getHeader("Host");
        $host_parts = string.split($host, ".");

        if ($path == "/verify" || $host_parts[0] == "verifier") {
          $cur_pool = connection.getVirtualServer();
          $new_pool = string.replace($cur_pool, "-idweb", "-idverifier");
          pool.use($new_pool);
        }
**** Associated
        Request Rules
        X-Forwarded-For    
        add ssl headers to requests    
        route-verifier-requests    
        persona-redirect    
         
        Response Rules
        sanitize version response headers    
        add HSTS response header    
*** Request Logging
**** enabled : yes
**** filename : /var/log/zeus/scl2-persona.services.mozilla.com.access_%{%Y-%m-%d-%H}t
**** format : Custom : %h %{Host}i %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" "%{Cookie}i" "ssl: %{SSLClientCipher}i" node_s:%T req_s:%R retries:%C
*** Connection Management
**** All defaults
* Catalogs... Monitors... Create new
** Name : HTTP __heartbeat__ deep check
** Type : HTTP
** Scope : node
** path : /__heartbeat__?deep=true
** status_regex : ^200$
** note : /__heartbeat__?deep=true must return 200



https://zlb1.pub.phx1.svc.mozilla.com:9090/
* Services... Traffic IP Group... Create new
** Name : phx1-persona
** IP Address : 63.245.217.134
** IP Distribution Mode : multi-hosted
*** IP : 239.100.1.10
* Services... Virtual Servers... Create new
** Name : phx1-persona
** Protocol : HTTP
** Port : 443
** Default Pool : phx1-idweb
** Listening on : phx1-persona (Traffic IP Group)
** Notes : *.persona.org
*** SSL Decryption
**** Loaded the www.persona.org-EV cert
**** ssl_decrypt : yes
**** default certificate : www.persona.org-EV
*** Rules
**** Created new rule "persona-redirect"
        $host = "login.persona.org";
        $path = http.getPath();
        $hostheader = http.getHostHeader();

        if ($path != "/__heartbeat__") {
           if ($hostheader == "persona.org" ||
               $hostheader == "browserid.org" ||
               $hostheader == "www.browserid.org") {
              $qs = http.getQueryString();
              if ($qs == "") {
                 http.redirect("https://" . $host . $path);
              } else {
                 http.redirect("https://" . $host . $path . "?" . $qs);
              }
           }
        }
**** Modified rule "route-verifier-requests" from
        $path = http.getPath();

        if ($path == "/verify") {
          $cur_pool = connection.getVirtualServer();
          $new_pool = string.replace($cur_pool, "-idweb", "-idverifier");
          pool.use($new_pool);
        }
to
        $path = http.getPath();
        $host = http.getHeader("Host");
        $host_parts = string.split($host, ".");

        if ($path == "/verify" || $host_parts[0] == "verifier") {
          $cur_pool = connection.getVirtualServer();
          $new_pool = string.replace($cur_pool, "-idweb", "-idverifier");
          pool.use($new_pool);
        }
**** Associated
        Request Rules
        X-Forwarded-For    
        add ssl headers to requests    
        route-verifier-requests    
        persona-redirect    
         
        Response Rules
        sanitize version response headers    
        add HSTS response header    
*** Request Logging
**** enabled : yes
**** filename : /var/log/zeus/phx1-persona.services.mozilla.com.access_%{%Y-%m-%d-%H}t
**** format : Custom : %h %{Host}i %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" "%{Cookie}i" "ssl: %{SSLClientCipher}i" node_s:%T req_s:%R retries:%C
*** Connection Management
**** All defaults
* Catalogs... Monitors... Create new
** Name : HTTP __heartbeat__ deep check
** Type : HTTP
** Scope : node
** path : /__heartbeat__?deep=true
** status_regex : ^200$
** note : /__heartbeat__?deep=true must return 200

Comment 31

[Gene Wood [:gene]]

Hey folks, sorry for the spam, I'll put further deployment-centric notes in the ChangeWindow wiki page : https://intranet.mozilla.org/Services/Ops/ChangeWindow_20120709

Comment 32

[James Bonacci [:jbonacci]]

Well this is out there, so let me just Resolve and Verify the ticket...