1202781 - Need treeherder failover

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[Sheeri Cabral [:sheeri]]

Need to failover treeherder production database to defragment as per bug 1190361. We will need 20 mins of prep time and 5 mins of actual database downtime for this.

Comment 1

[Sheeri Cabral [:sheeri]]

Adding pythian for informational purposes only, as they cannot yet access the load balancer to do the failover.

Comment 2

[Linda Ypulong [:unixfairy]]

Hal - we would prefer to do this before next TCW - can you socialize and see if that would be possible

Comment 3

[Ed Morley [:emorley]]

Opening bug since there is nothing confidential in here.

On the Treeherder side I'm pretty flexible about this, most of our tasks will resume with no problem. Ideally the same applies to TaskCluster submissions to us - they _should_ retry, but someone will have to confirm.

It would be best to do this at a quite time of course, and to give the sheriffs a heads up.

Comment 4

[Ed Morley [:emorley]]

s/quite/quiet/ even, and by that I mean lower push activity on the trees, so before the US awake on a weekday etc.

Comment 5

[Greg Arndt [:garndt]]

From what I understand of mozilla-taskcluster, it will attempt to retry submitting to treeherder.  I haven't dove much into the code that handles the retry, but I believe I have seen it attempt to submit multiple times.

Comment 6

[u429623]

There's no action releng needs to take if treeherder is down, so whatever time Ed & Sheeri set is fine.

Notification list should include releng, sheriffs, taskcluster, and treeherder teams.

Comment 7

[Sheeri Cabral [:sheeri]]

OK, other than whistlepig,I have:

email: [e-mail], [e-mail], [e-mail]

IRC: #releng #treeherder #moc

Hal - Any other notifications I've missed? 

Ed - "before US wakes up" includes east coast? What about at night? Does 7 or 8 pm Pacific work? (that'd be 10 or 11 pm Eastern, 2 or 3 am UTC, if I'm doing my time zone math properly). I'm thinking tomorrow evening (Thu, 7 or 8 pm Pacific) as a good time, that way we have Friday in case things go awry.

Comment 8

[u429623]

(In reply to Sheeri Cabral [:sheeri] from comment #7)
> OK, other than whistlepig,I have:
> 
> email: [e-mail], [e-mail],
> [e-mail]

That's sheriffs@m.o <= not c

add dev-tree-management@lists.m.o (picks up tree herder team)

Comment 9

[Linda Ypulong [:unixfairy]]

Waiting on final details before I approve from CAB - please needinfo me when set

Comment 10

[Sheeri Cabral [:sheeri]]

OK, thanks Hal, and waiting on Ed for confirmation of when it's OK.

Comment 11

[Ed Morley [:emorley]]

(In reply to Sheeri Cabral [:sheeri] from comment #7)
> Ed - "before US wakes up" includes east coast? What about at night? Does 7
> or 8 pm Pacific work? (that'd be 10 or 11 pm Eastern, 2 or 3 am UTC, if I'm
> doing my time zone math properly). I'm thinking tomorrow evening (Thu, 7 or
> 8 pm Pacific) as a good time, that way we have Friday in case things go awry.

Yeah that probably works too, though I'll be asleep then, so if you wanted someone from Treeherder to be around, you'll need to ask :camd :-)

Comment 12

[Sheeri Cabral [:sheeri]]

OK, will ping :camd. How about 7 or 8 pm Pacific on Monday, Sep 14th?

Comment 13

[Cameron Dawson [:camd]]

Sure, I can be around.  Please ping me that afternoon so I don't forget.  :)  I'm usually in the midst of bedtime for the little ones around then.  But I can be available.  I've got IRC on my phone and I'll be sure to check that

Comment 14

[Sheeri Cabral [:sheeri]]

Treeherder successfully failed over in the past 10 minutes. philor confirmed things were working.