695564 - Deploy account-portal rpm-2.9-3 (along with server-core rpm-2.7-2)

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Toby Elliott [:telliott]]

Release notes:

(account portal)
2.9 - 2011-10-18
================

Impacts:

- QA

Dependencies:

- python-services 2.7

Relevant changes:

- Cleanup for new config (requires 2.7)
- Bug 695540 - Oops hook can now get the crashid
- Bug 683395 - doesn't try to delete your data if you don't have a node


(server-core)
2.7 - 2011-10-18
================

Impacts:

- Ops
- QA

Relevant changes:

A whole lot. Reconfiguration of a whole bunch of libraries, including how we handle authentication, configuration and where utility functions live. Rewritten testing support libraries.

Any product that moves to this core should check all functionality to see if anything breaks.

Comment 1

[Toby Elliott [:telliott]]

QA plan:

As noted in the server core notes above, all account portal functionality should be checked. Two additional tests should be run for the two bugs above:

1) have ops set something in the back to cause a 503. Verify that the error page gives you a crashid to report back.

2) create an account, but don't get a node (or have ops remove your node from ldap. delete the account and verify that an error does not occur.

Comment 2

[Pete Fritchman [:petef]]

deployed in stage, https://stage-account.services.mozilla.com

two bugs so far:
* dinosaurs/error page does not include sync.css from the right path
* TB when changing password: http://petef.pastebin.mozilla.org/1359671

Comment 3

[Pete Fritchman [:petef]]

(In reply to Pete Fritchman [:petef] from comment #2)
> * TB when changing password: http://petef.pastebin.mozilla.org/1359671

after restarting sreg gunicorns, things are happier.

Comment 4

[James Bonacci [:jbonacci]]

QA has picked this up and will start working on the general AP functionality.

Comment 5

[Toby Elliott [:telliott]]

account-portal-2.9-3 and server-core-2.7-2 have the code to address the first problem in comment 2. Bug title modified appropriately.

Comment 6

[Pete Fritchman [:petef]]

new versions deployed, ready for QA.

Comment 7

[James Bonacci [:jbonacci]]

Pete I have no access to IRC at the moment. Will load be required to run tests in Stage?

Comment 8

[Pete Fritchman [:petef]]

(In reply to James Bonacci [:jbonacci] from comment #7)
> Pete I have no access to IRC at the moment. Will load be required to run
> tests in Stage?

nope, we don't have a loadtest for account portal.

Comment 9

[James Bonacci [:jbonacci]]

Actually, I meant for syncing new and current accounts before blasting them through AP....
;-)

Will be doing the normal sanity/smoke testing of this over the weekend, then hit the special tests this Monday.

Comment 10

[James Bonacci [:jbonacci]]

FYI:
ap1 nginx error log is showing a number of related errors between 6:52am ad 6:57am on 10/22.
(I was not doing any testing at this point)

related to
/var/html/accountportal
/var/html/accountportal/fonts/
/var/html/accountportal/images/
etc.

as in
...directory index of "/var/html/accountportal/images/" is forbidden...


for clients
client: 10.14.210.131
client: 10.14.210.132

Comment 11

[James Bonacci [:jbonacci]]

Need OPs to verify the deletion of a troublesome account on Stage.
Last info was the following:
User: ivuwkzrie6e47unuoldepdfqmsqc22a3
Node: stage-sync80.services.mozilla.com
for the login: jbonacci+test11

Got this message in stage-account:
We were unable to delete your data on the weave node. Don't worry, it's encrypted on the node and will be cleaned up shortly.
Thank you for using Mozilla Services. We have deleted your account.

Nothing unusual in the ap* nginx logs

Comment 12

[James Bonacci [:jbonacci]]

Further notes on Comment 10.
Forgot to mention all the Traceback information in /var/log/accountportal.log
on ap1 for the same timeframe (6:52 - 6:57)



Also noticed this for jbonacci+test11 (per Comment 11)
delete_data failure: 503 (url: https://stage-sync80.services.mozilla.com/1.1/ivuwkzrie6e47unuoldepdfqmsqc22a3/storage) "server issue: database marked as down"

Followed by:
2011-10-22 12:05:30,963 ERROR [accountportal] account deleted: ivuwkzrie6e47unuoldepdfqmsqc22a3
2011-10-22 12:05:30,963 ERROR [accountportal] account deleted: ivuwkzrie6e47unuoldepdfqmsqc22a3

Comment 13

[James Bonacci [:jbonacci]]

Tracebacks not related to AP code per Toby.
Moving on...

Comment 14

[James Bonacci [:jbonacci]]

Completed some extensive AP testing. Everything looks good so far.
Will continue with items in Comment 1 with OPs on Monday.

Comment 15

[James Bonacci [:jbonacci]]

OK, Pete and I verified the fixes listed in Comment 1.
Scheduling Prod deployment for Tuesday, 10/25.

Comment 16

[Pete Fritchman [:petef]]

deployed

Comment 17

[Pete Fritchman [:petef]]

fun notes:

we were going to roll this back because of account deletion failing, but i noticed in logs it's been failing.

turns out sreg had a too-new version of python26-webob (1.0.8). even though we're pinning 1.0.7 and building that, yum hasn't been installing because it's "smart" and won't downgrade.

downgraded python26-webob to 1.0.7 manually on sreg, things are happy.

Comment 18

[James Bonacci [:jbonacci]]

Verified in Production just now (PHX1), that I get this message:
Thank you for using Mozilla Services. We have deleted your account.

Instead of this one:

Oh dear.
Looks like one of the dinosaurs escaped again. We keep them away from the data, so that should be safe.
Please try again later when we've wrestled him back onto the treadmill.
If you want to report a sighting, please make sure to use their id tag: BLAH BLAH BLAH. Thanks!

Will wait for final review from Pete before marking this as Verified.