As a security precaution, we have turned on the setting "Require API key authentication for API requests" for everyone. If this has broken something, please contact bugzilla-admin@mozilla.org
Last Comment Bug 524794 - Upgrade surf to RHEL5
: Upgrade surf to RHEL5
Status: RESOLVED FIXED
Tuesday Nov. 17th 6am-9am PST
:
Product: mozilla.org Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All Other
: -- enhancement (vote)
: ---
Assigned To: Shyam Mani [:fox2mike]
: matthew zeier [:mrz]
:
Mentors:
Depends on:
Blocks: 529313 529375
  Show dependency treegraph
 
Reported: 2009-10-27 15:13 PDT by Dave Miller [:justdave] (justdave@bugzilla.org)
Modified: 2015-03-12 08:17 PDT (History)
12 users (show)
justdave: needs‑downtime+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description User image Dave Miller [:justdave] (justdave@bugzilla.org) 2009-10-27 15:13:21 PDT
Surf got left out of our attempts to upgrade everything to RHEL5 a while back, for the sole reason that we were intending to replace the machine with dm-stage01/02 and figured it would just pick up the upgrade as part of being decommissioned or reused.

However, dm-stage01/02 haven't panned out yet, and it may be a while, so there's no reason to hold off on it.  This is one of the last machines we have left still on RHEL4.

This would require about an hour of downtime for stage.mozilla.org, and has the potential for their to be weird small problems cropping up over the next day or so after the upgrade while we fine-tune things that may work "differently" in RHEL5.
Comment 1 User image Dave Miller [:justdave] (justdave@bugzilla.org) 2009-10-27 15:18:47 PDT
Benefits of upgrading:

1) Our new backup stuff may or may not work on RHEL4, and will probably require hacking to get it to work there, but we know it works on RHEL5.

2) On RHEL5 we'll have the capability to lock down what you can do in a shell on stage.mozilla.org (so that you can scp, rsync, sftp, etc, but you can't get an interactive shell on a normal user account) which is something we've wanted to do for a while.

3) Many general performance improvements and easier package management.
Comment 2 User image Nick Thomas (PTO 23-27 January, UTC+13) [:nthomas] 2009-10-27 15:52:09 PDT
From a RelEng point of view we should pick a downtime away from any releases, so the next couple of weeks would be fine. I can be available to help verify that it's working afterward, assuming you're thinking of PDT evening downtime.
Comment 3 User image Ben Hearsum (:bhearsum) 2009-10-27 16:15:09 PDT
(In reply to comment #1)
> Benefits of upgrading:
> 
> 1) Our new backup stuff may or may not work on RHEL4, and will probably require
> hacking to get it to work there, but we know it works on RHEL5.
> 
> 2) On RHEL5 we'll have the capability to lock down what you can do in a shell
> on stage.mozilla.org (so that you can scp, rsync, sftp, etc, but you can't get
> an interactive shell on a normal user account) which is something we've wanted
> to do for a while.

Can we please make sure this is a separate item? We've been pretty good about not adding new things that depend on the shell, but there's a ton of existing stuff that would need to be changed before we can do this on ffxbld/cltbld accounts.
Comment 4 User image Dave Miller [:justdave] (justdave@bugzilla.org) 2009-10-27 17:23:56 PDT
It's already a separate item.  It's not part of this, this is just a prerequisite for it.
Comment 5 User image Ben Hearsum (:bhearsum) 2009-10-28 05:13:10 PDT
(In reply to comment #4)
> It's already a separate item.  It's not part of this, this is just a
> prerequisite for it.

Thanks for the clarification
Comment 6 User image Robert Kaiser 2009-10-28 09:07:15 PDT
(In reply to comment #3)
> [...], but there's a ton of existing
> stuff that would need to be changed before we can do this on ffxbld/cltbld
> accounts.

As a note, seabld is one of those accounts as well, as well as the Thunderbird one (tbirdbld?) - is there a bug for that further step?
Comment 7 User image Dave Miller [:justdave] (justdave@bugzilla.org) 2009-10-28 22:45:25 PDT
(In reply to comment #6)
> As a note, seabld is one of those accounts as well, as well as the Thunderbird
> one (tbirdbld?) - is there a bug for that further step?

I'll file one when we decide to go ahead with it (and make sure it gets trumpeted loudly well in advance on dev-planning and planet and so forth).  The existing bug kind of has it lumped in with the stage.m.o rebuild, which is indefinitely stalled at the moment.
Comment 8 User image matthew zeier [:mrz] 2009-11-04 14:49:24 PST
build, when's a good time to schedule this?  pass over when ready.
Comment 9 User image Chris Cooper [:coop] 2009-11-13 09:20:52 PST
I sent mail out to releng last night, and due to overwhelming ambivalence, we've decided upon:

Tuesday Nov. 17th 6am-9am PST

with a back-up choice of:

Thursday Nov. 19th 6am-9am PST

Let us know if that doesn't work for IT.
Comment 10 User image Dave Miller [:justdave] (justdave@bugzilla.org) 2009-11-14 09:33:47 PST
works for me.
Comment 11 User image Chris Cooper [:coop] 2009-11-16 08:53:29 PST
(In reply to comment #10)
> works for me.

Do you want me to send out a downtime notice for this, or were you planning to?
Comment 12 User image Shyam Mani [:fox2mike] 2009-11-16 20:47:05 PST
I'll be doing the actual upgrade work. I plan to start at 0600 PST and will be in the usual channels in case people need to get in touch :)
Comment 13 User image Dave Miller [:justdave] (justdave@bugzilla.org) 2009-11-16 21:11:40 PST
Downtime notice has been sent to the usual places.  Separate copy with different wording specific to the mirror network has been sent to the mirrors list.
Comment 14 User image Dave Miller [:justdave] (justdave@bugzilla.org) 2009-11-17 10:16:56 PST
status: NFS mounts failed to happen when the machine was booted after the upgrade.  Unfortunately, this didn't get caught until after the rsync hubs had all rsynced out the now-empty docroot, so the releases site is kind of hosed at the moment.  Working on trying to get the nfs mounts back online.
Comment 15 User image Shyam Mani [:fox2mike] 2009-11-17 21:06:04 PST
I think we can safely call this done. Thanks to Dave for the help!

Note You need to log in before you can comment on or make changes to this bug.