support1.webapp.phx1.mozilla.com crashed

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: ericz, Assigned: ericz)

Tracking

Details

(Assignee)

Description

4 years ago
support1.webapp.phx1.mozilla.com crashed and has outdated firmware.  If we're ever likely to get support from HP, it'll need the firmware upgraded.  There are additionally 197 packages that yum would be happy to upgrade.  Can we reboot this box to apply the firmware and package updates?
(Assignee)

Updated

4 years ago
Flags: needinfo?(dmaher)
Is it just support1 that needs to be updated, or all of the support[1-5] as well ?
Flags: needinfo?(dmaher) → needinfo?(eziegenhorn)
(Assignee)

Comment 2

4 years ago
Likely they are all in similar shape, though I haven't checked.  I'd be happy to do them all in a rolling fashion if that's possible.
Flags: needinfo?(eziegenhorn)
Yes, these firmware and system updates are important, so these machines should definitely be processed in a rolling fashion (as you've indicated).  Please co-ordinate with :r1cky in #sumo, as I'm sure he would very much like to be kept apprised of the operation.
(Assignee)

Comment 4

4 years ago
Coordinated with r1cky and upgraded BIOS and RAID firmware as well as all packaged on support[1-5].
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(In reply to Eric Ziegenhorn :ericz from comment #4)
> Coordinated with r1cky and upgraded BIOS and RAID firmware as well as all
> packaged on support[1-5].

Upgrading all packages damaged puppet, which is no longer able to run due to the upgrades. Was 'yum-wrapper' used to do the upgrades?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Puppet has been repaired, but I've disabled it until someone can run it at a safe time - there's firmware updates, collectd, dnsmasq, and hp_utils updates pending, and I'm not comfortable pushing those changes without oncall involvement.

> Puppet has been disabled by rsoderberg at 2014-01-21 05:38 until 2015-01-21 05:38 with the following message: disabled until oncall can run this at a safe time, bug 949159

The next step is to run puppet on this host and then confirm that it's still working in whatever capacity it serves for us. I suspect there will be additional firmware updates to apply, also verify that the HP utilities are working correctly afterwards.
(Assignee)

Comment 7

4 years ago
No I didn't think of yum-wrapper.  Also, I thought we had pinned all necessary packages such that yum did the right thing.  I'll remember that going forward, thanks for fixing puppet on support1.  I re-enabled puppet and it looks good.
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
Unfortunately it turns out to be extremely difficult to enforce puppet version pins in yum, which is why we created yum-wrapper. Someday we'll understand how to do it on the yum side, but until then, the wrapper is a stopgap fix.
(In reply to Richard Soderberg [:atoll] from comment #8)
> Unfortunately it turns out to be extremely difficult to enforce puppet
> version pins in yum, which is why we created yum-wrapper. Someday we'll
> understand how to do it on the yum side, but until then, the wrapper is a
> stopgap fix.

In fact this is why, on the Elasticsearch servers, I enforce a Yum _exclude_ on the elasticsearch package itself.  This is the only way that I know of to avoid accidentally "upgrading" the package via yum...
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.