1097875 - Need to failover for patching - treeherder

Status: RESOLVED FIXED

(Infrastructure & Operations :: Change Requests, task)

Description

[Sheeri Cabral [:sheeri]]

Need to patch:

treeherder1.db.scl3.mozilla.com

so we need to failover, patch and fail back (probably about 30 mins to patch, but failover causes less than a minute-long outage)

Owners: :fubar, :mdoglio, :camd

When can this be done?

Comment 1

[Kendall Libby [:fubar] (he/him)]

Work on treeherder has the potential to close trees; NI on jgriffin to schedule.

Comment 2

[Kendall Libby [:fubar] (he/him)]

(I suspect EST morning will be best time of day, based on a convo we just had with sheriffs regarding when to push code changes to prod.)

Comment 3

[Jonathan Griffin (:jgriffin)]

I agree EST morning (not on a Friday) is probably the best time; ed or Ryan, do you disagree?

Comment 4

[Ryan VanderMeulen [:RyanVM]]

++

Comment 5

[Ed Morley [:emorley]]

wfm

Comment 6

[Sheeri Cabral [:sheeri]]

OK, I will fail over tomorrow morning at 7 am Pacific, and fail back Monday at 7 am Pacific. I assume silence = acceptance. Watch this bug for updates.

Comment 7

[Kendall Libby [:fubar] (he/him)]

Before starting, can you drop a ping in #treeherder? It is especially helpful to the sheriffs to know when work's being done on the infra. Thx!

Comment 8

[Sheeri Cabral [:sheeri]]

Yes! I missed today's window, sadly. (calendar reminder didn't remind me!)

Comment 9

[Shyam Mani [:fox2mike]]

Approved by the CAB for the date/time mentioned in the CAB spreadsheet.

Comment 10

[Rick Bryce [:rbryce]]

With sheeri's & treeherder's permission. I failed over and failed back to the original master/slave config, after upgrading both mysql servers.