Need to failover for patching - treeherder

RESOLVED FIXED

Status

Infrastructure & Operations
Change Requests
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: sheeri, Assigned: rbryce)

Tracking

Details

(Whiteboard: [data: db_mig upgrade])

(Reporter)

Description

3 years ago
Need to patch:

treeherder1.db.scl3.mozilla.com

so we need to failover, patch and fail back (probably about 30 mins to patch, but failover causes less than a minute-long outage)

Owners: :fubar, :mdoglio, :camd

When can this be done?
Flags: cab-review?
Work on treeherder has the potential to close trees; NI on jgriffin to schedule.
Flags: needinfo?(jgriffin)
(I suspect EST morning will be best time of day, based on a convo we just had with sheriffs regarding when to push code changes to prod.)
I agree EST morning (not on a Friday) is probably the best time; ed or Ryan, do you disagree?
Flags: needinfo?(ryanvm)
Flags: needinfo?(jgriffin)
Flags: needinfo?(emorley)
++
Flags: needinfo?(ryanvm)
wfm
Flags: needinfo?(emorley)
(Reporter)

Comment 6

3 years ago
OK, I will fail over tomorrow morning at 7 am Pacific, and fail back Monday at 7 am Pacific. I assume silence = acceptance. Watch this bug for updates.
Before starting, can you drop a ping in #treeherder? It is especially helpful to the sheriffs to know when work's being done on the infra. Thx!
(Reporter)

Comment 8

3 years ago
Yes! I missed today's window, sadly. (calendar reminder didn't remind me!)
Approved by the CAB for the date/time mentioned in the CAB spreadsheet.
Flags: cab-review? → cab-review+
(Assignee)

Comment 10

3 years ago
With sheeri's & treeherder's permission. I failed over and failed back to the original master/slave config, after upgrading both mysql servers.
Assignee: server-ops → rbryce
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Whiteboard: [data: p2v upgrade]
(Reporter)

Updated

3 years ago
Whiteboard: [data: p2v upgrade] → [data: db_mig upgrade]

Updated

2 years ago
Cab Review: --- → approved
Flags: cab-review+
Group: infra
You need to log in before you can comment on or make changes to this bug.