1093828 - Add a site-wide banner to warn users that the database is read-only

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: MozTrap, defect, P2)

Description

[Matt Brandt [:mbrandt]]

In the event the database reverts to a read-only state a site-wide banner should be displayed to warns users of the problem, the message should include who to contact.

Example scenario - recently an automated fail over in our data center occurred, the db failed over to a secondary master that defaults to read-only. From outward appearances the site appeared to be online but produced a 500 error when users tried to save test results.

Comment 1

[Stephen Donner [:stephend] Not actively reading bugmail]

Peter, might I cajole you somehow into looking into fixing this? :-)  (For context, came out of a post-mortem, and was identified as being really helpful to both end-users and on-call/sys/DB-admins, alike.)

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

(Or, Bob, feel free to grab this and take a look, too!)

Comment 3

[Peter Bengtsson [:peterbe]]

I'm actually not familiar with the read-only mode. Is this a thing and all we need to do is build the banner or does it need the read-only mode functionality too?

Comment 4

[Bob Silverberg [:bsilverberg]]

I wouldn't know where to start with this either, but I could try if Peter doesn't have time (and has advice). Is there a way for us to know that the database has been put into read-only mode?

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

cturra or sheeri, can you help with comment 3 / comment 4?  Thanks!

Comment 6

[Sheeri Cabral [:sheeri]]

you can query for "show variables like 'read_only' " and that will say ON or OFF  - so, read_only=ON would be when the db is read only.

Comment 7

[Bob Silverberg [:bsilverberg]]

Thanks Sheeri. I will give that a try.

Comment 8

[Bob Silverberg [:bsilverberg]]

Attachment: Link to Github pull-request: https://github.com/mozilla/moztrap/pull/77

Peter, I have done most of what is required for this, but I'm not sure if it's suitable for Moztrap. I basically borrowed code from Kitsune, so the styling may need to be changed. Also, I haven't implemented the Javascript to close the dialog because I find the organization and implementation of the JS in Moztrap to be very confusing. I was hoping that you might have some familiarity with it and be able to collaborate with me on this patch to add the required JS.

Comment 9

[Bob Silverberg [:bsilverberg]]

Matt, I have prepared a PR for this that mostly works and have asked Peter for review and some help. I think we may need to wordsmith the message as well. Right now it just says:

"There is a temporary problem with the Moztrap database that has put it into a read-only state.
You may browse information on the site but no changes will be saved."

You mentioned including contact information. If you can provide a preferred paragraph I can change the code to display whatever you think is appropriate.

Comment 10

[Matt Brandt [:mbrandt]]

Let's got with:

"There is a temporary problem and Moztrap has been placed into a read-only state. You may browse information on the site but changes will not be saved. Please contact us in #moztrap for more information."

Comment 11

[Peter Bengtsson [:peterbe]]

Comment on attachment 8522291 [details] [review]
Link to Github pull-request: https://github.com/mozilla/moztrap/pull/77

let's use GitHub instead to review the code.

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

With the changes now landed on master and pushed to dev, can we work with Sheeri, Peter/Bob, to test that the read-only banner shows up?

Comment 13

[Sheeri Cabral [:sheeri]]

Sure, just let me know when. I live in #data if you can't find me elsewhere on IRC. I am in Eastern time today and tomorrow and in Portland next week.

Comment 14

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Sheeri Cabral [:sheeri] from comment #13)
> Sure, just let me know when. I live in #data if you can't find me elsewhere
> on IRC. I am in Eastern time today and tomorrow and in Portland next week.

Sorry for the delay -- been a hectic few weeks :-(  I or someone from my team will find you this week, and try to have this tested before we (all?) take off soon for the holidays!

Comment 15

[Sheeri Cabral [:sheeri]]

About testing - Right now, the moztrap DB is on a shared server with many other dbs (one mysql instance, many dbs). Also, "read_only" is a server-wide setting, so if we test by changing "read_only" we will affect others.


Another way would be to point moztrap's "write" handlers to the RO VIP, and see what happens. That's what I'd recommend. (Conveniently there's no coordination with the data team needed either!)

Comment 16

[Matt Brandt [:mbrandt]]

camd/peterbe -- if you can find time, can we do this -- comment 15?

Comment 17

[Peter Bengtsson [:peterbe]]

(In reply to Sheeri Cabral [:sheeri] from comment #15)
> About testing - Right now, the moztrap DB is on a shared server with many
> other dbs (one mysql instance, many dbs). Also, "read_only" is a server-wide
> setting, so if we test by changing "read_only" we will affect others.
> 
> 
> Another way would be to point moztrap's "write" handlers to the RO VIP, and
> see what happens. That's what I'd recommend. (Conveniently there's no
> coordination with the data team needed either!)

If it's easy to do, let's do it. The dev server gets virtually no traffic.

Comment 18

[Cameron Dawson [:camd]]

Yep, sounds good to me.  Sheeri-- would you be up for doing this sometime tomorrow or Friday, maybe?  Dev is already running this code.

Comment 19

[Cameron Dawson [:camd]]

I should clarify: please ping peter, matt or I on irc to help test it.  :)

Comment 20

[Sheeri Cabral [:sheeri]]

Oh, we can definitely do this whenever. But this is an app-level thing - changing the app's config file (or wherever it's stored) FROM "db-rw-virtualIP" to "db-ro-virtualIP".

I'll see if I can figure out where this is stored so I can help you test it out, but I think this is a webops task usually.

Comment 21

[Philippe M. Chiasson (:gozer)]

(In reply to Sheeri Cabral [:sheeri] from comment #20)
> Oh, we can definitely do this whenever. But this is an app-level thing -
> changing the app's config file (or wherever it's stored) FROM
> "db-rw-virtualIP" to "db-ro-virtualIP".
> 
> I'll see if I can figure out where this is stored so I can help you test it
> out, but I think this is a webops task usually.

I can change that easily, moztrap lives in the generic cluster, and the dev config is:

genericadm.private.phx1.mozilla.com:/data/genericrhel6-dev/src/moztrap-dev.allizom.org/moz-trap/moztrap/settings/local.py

Should I go ahead and make the change now ?

Comment 22

[Philippe M. Chiasson (:gozer)]

Just read back the comments, so I decided to go ahead and turn dev read-only by pointing it at:
 - dev-zeus-ro.db.phx1.mozilla.com

Comment 23

[Sheeri Cabral [:sheeri]]

https://moztrap-dev.allizom.org/results/runs/ there is a banner there....

Comment 24

[Cameron Dawson [:camd]]

We tested it this morning, and the banner DOES show.  But, for both sheeri and I, not until we did a shift-reload of the page.  We may want to change the implementation so that it adds a suffix to the name like:
``MozTrap-dev - READ ONLY``  Where the read-only is red or something.  Hovering tells you why?  Just a thought.

Comment 25

[Cameron Dawson [:camd]]

fwiw: Sheeri and I decided to leave dev as read-only so it could be iterated on faster.  She says we can just ping her in IRC when we want it changed back.

Comment 26

[Peter Bengtsson [:peterbe]]

Regarding the caching, it seems to be in the browser only. When I run this: https://gist.github.com/peterbe/7060754a88c8d4aa07e9 it never fails. The banner is always there. 
If it got caught in Zeus that would not be so predictable. 

Grr! Why is the browser caching it?!

Comment 27

[Sheeri Cabral [:sheeri]]

peterbe says it should be more like wikipedia's fundraiser that shows up on top and doesn't let you close it, but still lets you do work. Specifically for the scenario that if you close it but come back 5 mins later and it doesn't show up, you don't know if things are back to normal or if it's still read-only but it remembers you X'd it out.

Comment 28

[Cameron Dawson [:camd]]

Hey Bob-- Would you be up for making this modification?

Comment 29

[Bob Silverberg [:bsilverberg]]

camd: I would, but I am on vacation until the new year, if it can wait until then. Leaving the needinfo on as a reminder.

Comment 30

[Bob Silverberg [:bsilverberg]]

Attachment: Link to Github pull-request: https://github.com/mozilla/moztrap/pull/85

This is a PR to address not permanently dismissing the banner

Comment 31

[Bob Silverberg [:bsilverberg]]

I believe the work for this was done and deployed. Can anyone comment on that? Camd? Peterbe?

Comment 32

[Peter Bengtsson [:peterbe]]

We did land this in master after all. And even though there might be some imperfections I'm sure it's a bunch better than nothing and besides it rarely happens (that mysql goes into read-only mode).

Feel free to re-open camd if you think it needs more work.

Comment 33

[Cameron Dawson [:camd]]

Agreed, yeah.  This was landed.  Sorry it was never resolved.  Thanks peterbe!