804255 - Make sure mysql master failover for AMO/Marketplace does not disrupt login

Status: RESOLVED WONTFIX

(Data & BI Services Team :: DB: MySQL, task)

Description

[Kumar McMillan [:kumar]]

When sheeri triggered AMO to failover its master DBs we were unable to log in to Marketplace with Persona without clearing cookies. The server was returning a 403. We scrambled to fix it so I'm not sure the actual cause was. Maybe it was something session related? Let's try it in a more controlled environment to prevent it from happening in a real failover.

Comment 1

[Wil Clouser [:clouserw]]

Does this get fixed by bug 795363?

Comment 2

[Kumar McMillan [:kumar]]

I'm not sure if sessions were the actual cause here. In theory, failing over to a new master shouldn't be a problem for sessions if it has the same data :/ So maybe something else was causing the error. If we can reproduce it then we can check what was causing the 403. It persisted until we cleared cookies (many of us hit the error).

Comment 3

[Kumar McMillan [:kumar]]

I forgot to mention that we were hitting this exception:
(1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')

Details: http://sentry.dmz.phx1.mozilla.com/marketplace/group/2722/

Maybe that put sessions in a broken state.

Comment 4

[Sheeri Cabral [:sheeri]]

That could be, and that was my fault for not updating puppet before actually changing the masters over in the load balancer. Still, we should test this - it's easy enough to do, we just need 10 mins or so, and it can be done at any time we're all ready to test (not urgent, but it's a good idea).

The --read-only issue would happen in case of an actual failover, too, so we should revisit that and maybe make the failover world-writable.

Comment 5

[Sheeri Cabral [:sheeri]]

(er, not world-writable, but read_only set to OFF).

Comment 6

[Kumar McMillan [:kumar]]

+1 for testing this on stage (or somewhere like that).

Comment 7

[Wil Clouser [:clouserw]]

https://wiki.mozilla.org/Marketplace/HAtesting will probably be of interest to people following this bug

Comment 8

[Matt Basta [:basta]]

This might not be as big of an issue with Fireplace. The API's SSA doesn't hit the database to authenticate the user, so users authenticated before a failure won't get 404'd or something equally catastrophic.

Obviously this doesn't apply to the devhub/reviewer tools/admin stuff.

Comment 9

[Sheeri Cabral [:sheeri]]

Will you test failover when you patch for the Ghost vulnerability (glibc)? Do you need any assistance with that from my team?

Comment 10

[Sheeri Cabral [:sheeri]]

Data team no longer manages marketplace/amo production, so this isn't in our wheelhouse.