Closed Bug 335151 Opened 14 years ago Closed 13 years ago

Upgrade b.m.o to 2.23.4

Categories

(bugzilla.mozilla.org :: General, defect, P2, major)

Tracking

()

VERIFIED FIXED

People

(Reporter: LpSolit, Assigned: justdave)

References

()

Details

(Whiteboard: Completed 26 December 2006 10:21pm PST)

Now that 2.22 has been released, it makes real sense to upgrade. There are some new features I would like to use which are not available on the 2.20 branch. ;)
Wouldn't we all...
Alias: bmo-upgrade
Priority: -- → P3
Severity: normal → major
Priority: P3 → P2
i'd like to have bug 314056 backported for bmo with the bmo update :(.
i'd like to add bug 313571 to the list, as not having it results in:

Saved Searches:
My Bugs | Camino | check1.8.0.2 | check1.8.0.2 | check1.8.1 | check1.8.1 | patched | QA | Ready | Ready | Ready (Locked) | Ready (Locked) | requests | requests | security‑month | test | test | timeless
yes, somebody with the powers to be, please update the installation.  Thank you.
We're in the middle of shuffling a bunch of the webtools stuff around, this will hopefully come soon afterwards, but no guarantees.  Commenting here saying "please upgrade" isn't likely to help encourage it any faster than it's already coming. :)
I'd like to have bug 274802 backported, also.
(In reply to comment #6)
> I'd like to have bug 274802 backported, also.

Two seconds after I sent this, I noticed that this would not be very possible, as that bug requires so many prerequisites. :/
/me suggests to wait for 2.22.1 before upgrading, to catch security fixes.
From what justdave said on IRC, he will upgrade b.m.o to 3.0 RC1 instead of 2.22.1.
Summary: Upgrade b.m.o to 2.22 → Upgrade b.m.o to 3.0 RC1
OK, here's the downtime analysis:

  15 min - pre-upgrade database backup
  20 (30) min - apache, mod_perl, mod_ssl, various perl module upgrades, upgrade
                the Bugzilla code itself (this can be started while the backup
                is running, so I'm counting it as a 10 minute overlap)
  80 min - 2.20 -> 3.0 schema conversion
  25 min - multiple charsets -> utf8 charset conversion
  15 min - post-upgrade database backup
   0 (10) min - apache config for mod_perl (can be done during the 2nd backup)
--------
 155 min - total downtime if all goes well.

Built-in troubleshooting time:

  45 min - want to allow some time to attempt to fix minor things that might be
           broken before resorting to restoring backups

In case of reversion:

  40 min - restore backups of database and bugzilla code (the newer apache and
           perl modules are generally well-tested upstream and should still run
           the older Bugzilla code, so attempting to revert those is not
           foreseen as potentially necessary)

========
 240 min (4 hours) - total downtime window



FWIW, here's the distribution of non-ASCII non-UTF8 data currently in bmo across all text fields in all tables:

      2 euc-jp
      3 iso-8859-8
     11 cp855
     20 utf-8-strict
     24 big5-eten
     31 cp866
     34 cp1253
     41 iso-8859-7
     50 iso-8859-5
     64 koi8-r
    155 cp1251
    273 cp1255
    370 shiftjis
    704 Failed to detect
  21338 cp1252 or iso-8859-1

Those 704 "failed to detect" items are cause for concern.  There's also a chance that some items which did detect may have detected wrong.

There's a little bit more pre-upgrade work to be done before we can set a date (converting our local hacks to continue working in 3.0 mainly).  Should have a date soon.
QA Contact: myk → justin
sometime in the next day or so I'll have an upgraded copy of bmo running with email disabled on another URL for people to play with.
https://bugzilla-stage.mozilla.org/ is the test install.  Mail is disabled so you can play without it mailing people.

It does NOT have any of our local customizations applied yet, it's cvs-tip Bugzilla as of last week just running on a backup copy of the production database from a week or two ago.

Spreadsheet tracking the process of porting the patches forward is at http://spreadsheets.google.com/ccc?key=p-AiBqJKo-QywsXTT5r2bQw

If you'd like to help port patches be my guest. :)  The patches are all public (urls are at the top of the spreadsheet).
Whiteboard: Tentatively scheduled for 26 December 2006
Before upgrading to 3.0 RC1, you should fix all errors reported by sanitycheck.cgi. The list is *very* long, and you could get unexpected errors if you don't do it first.
Blocks: 362778
Blocks: 313910
No longer depends on: 313910
The times in comment 10 were based on running the upgrade scripts in a VM that was talking to a database running on a different VM.  Running it against a copy of bmo in a different directory on the production server talking to a backup copy of the database also on the production DB server was significantly faster.  The schema upgrade portion is 20 minutes shorter, and the charset conversion is 15 minutes shorter.  That said, I don't think I'll change the estimate. :)  Can always use the extra time for cleanup or troubleshooting if anything goes wrong.

What do we want to do about the charset conversion?  There's 716 fields in the database with an unguessable charset now.  Most of them are comments and summaries (and changes to summaries in the activity table).  I can get a list of the unmatched fields if anyone wants to try to decipher them (we can feed it per-item overrides if we find something wrong, and set a default charset to assume if it can't guess, which is probably windows-1252).

I'm thinking at this point that we just use the windows-1252 fallback and keep a backup of the database around to restore from if anyone finds anything strange.  It's going to be hell to try to manually figure out the several hundred unguessed entries.
Oh, turns out there's an additional step to the charset conversion.  After the recoding is done, we have to go back and convert the table storage in the database so that MySQL actually knows that the data is UTF-8 (so sorting happens correctly and so forth).  That process just took 55 minutes to run.  So we're up 20 minutes now instead of down 35. :)
On the other hand, if some comments are encoded in an unknown charset, they are probably not displayed correctly in your browser and so there is not really dataloss here as we cannot display them in a readable manner anyway. I wouldn't worry too much about them.
I forgot -- we get a little time back because we no longer have to upgrade Apache and perl modules and so forth.  All of the prereqs are in place on the production server already as of this last Tuesday.

So here's what our downtime schedule looks like now:

  15 min - pre-upgrade database backup
  60 min - 2.20 -> 3.0 schema conversion
  10 min - multiple charsets -> utf8 charset conversion
  60 min - iso-8859-1 -> utf8 MySQL storage conversion
  15 min - post-upgrade database backup
   0 (10) min - apache config for mod_perl (can be done during the 2nd backup)
--------
 160 min - total downtime if all goes well.
 (2h40m)

Built-in troubleshooting time:

 160 min - want to allow some time to attempt to fix minor things that might be
 (2h40m)   broken before resorting to restoring backups

In case of reversion:

  40 min - restore backups of database and bugzilla code (the newer apache and
           perl modules are generally well-tested upstream and should still run
           the older Bugzilla code, so attempting to revert those is not
           foreseen as potentially necessary)

========
 360 min (6 hours) - total downtime window

We've got a lot of recovery time built into the advertised outage window.  From how things are looking in testing it's probable that we won't need it.  So we're probably actually looking at about 3 hours for the full upgrade.
Whiteboard: Tentatively scheduled for 26 December 2006 → Scheduled for 26 December 2006 6pm PST
All done!
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Yay.
Status: RESOLVED → VERIFIED
3.0rc1 isn't released yet, mostly as a result of things we found while staging this upgrade.  But we've upgraded anyway (shoved a LOT of fixes upstream in the last week).

Regressions are being tracked on bug 365063.
Summary: Upgrade b.m.o to 3.0 RC1 → Upgrade b.m.o to 2.23.4
Whiteboard: Scheduled for 26 December 2006 6pm PST → Completed 26 December 2006 10:21pm PST
QA Contact: justin → reed
No longer blocks: 329701
No longer depends on: 353690
Alias: bmo-upgrade
Component: Bugzilla: Other b.m.o Issues → General
Product: mozilla.org → bugzilla.mozilla.org
You need to log in before you can comment on or make changes to this bug.