335151 - Upgrade b.m.o to 2.23.4

Status: VERIFIED FIXED

(bugzilla.mozilla.org :: General, defect, P2)

Description

[Frédéric Buclin]

Now that 2.22 has been released, it makes real sense to upgrade. There are some new features I would like to use which are not available on the 2.20 branch. ;)

Comment 1

[Dave Miller [:justdave]]

Wouldn't we all...

Comment 2

[timeless]

i'd like to have bug 314056 backported for bmo with the bmo update :(.

Comment 3

[timeless]

i'd like to add bug 313571 to the list, as not having it results in:

Saved Searches:
My Bugs | Camino | check1.8.0.2 | check1.8.0.2 | check1.8.1 | check1.8.1 | patched | QA | Ready | Ready | Ready (Locked) | Ready (Locked) | requests | requests | security‑month | test | test | timeless

Comment 4

[Rolf Leggewie]

yes, somebody with the powers to be, please update the installation.  Thank you.

Comment 5

[Dave Miller [:justdave]]

We're in the middle of shuffling a bunch of the webtools stuff around, this will hopefully come soon afterwards, but no guarantees.  Commenting here saying "please upgrade" isn't likely to help encourage it any faster than it's already coming. :)

Comment 6

[Reed Loden [:reed]]

I'd like to have bug 274802 backported, also.

Comment 7

[Reed Loden [:reed]]

(In reply to comment #6)
> I'd like to have bug 274802 backported, also.

Two seconds after I sent this, I noticed that this would not be very possible, as that bug requires so many prerequisites. :/

Comment 8

[Frédéric Buclin]

/me suggests to wait for 2.22.1 before upgrading, to catch security fixes.

Comment 9

[Frédéric Buclin]

From what justdave said on IRC, he will upgrade b.m.o to 3.0 RC1 instead of 2.22.1.

Comment 10

[Dave Miller [:justdave]]

OK, here's the downtime analysis:

  15 min - pre-upgrade database backup
  20 (30) min - apache, mod_perl, mod_ssl, various perl module upgrades, upgrade
                the Bugzilla code itself (this can be started while the backup
                is running, so I'm counting it as a 10 minute overlap)
  80 min - 2.20 -> 3.0 schema conversion
  25 min - multiple charsets -> utf8 charset conversion
  15 min - post-upgrade database backup
   0 (10) min - apache config for mod_perl (can be done during the 2nd backup)
--------
 155 min - total downtime if all goes well.

Built-in troubleshooting time:

  45 min - want to allow some time to attempt to fix minor things that might be
           broken before resorting to restoring backups

In case of reversion:

  40 min - restore backups of database and bugzilla code (the newer apache and
           perl modules are generally well-tested upstream and should still run
           the older Bugzilla code, so attempting to revert those is not
           foreseen as potentially necessary)

========
 240 min (4 hours) - total downtime window



FWIW, here's the distribution of non-ASCII non-UTF8 data currently in bmo across all text fields in all tables:

      2 euc-jp
      3 iso-8859-8
     11 cp855
     20 utf-8-strict
     24 big5-eten
     31 cp866
     34 cp1253
     41 iso-8859-7
     50 iso-8859-5
     64 koi8-r
    155 cp1251
    273 cp1255
    370 shiftjis
    704 Failed to detect
  21338 cp1252 or iso-8859-1

Those 704 "failed to detect" items are cause for concern.  There's also a chance that some items which did detect may have detected wrong.

There's a little bit more pre-upgrade work to be done before we can set a date (converting our local hacks to continue working in 3.0 mainly).  Should have a date soon.

Comment 11

[Dave Miller [:justdave]]

sometime in the next day or so I'll have an upgraded copy of bmo running with email disabled on another URL for people to play with.

Comment 12

[Dave Miller [:justdave]]

https://bugzilla-stage.mozilla.org/ is the test install.  Mail is disabled so you can play without it mailing people.

It does NOT have any of our local customizations applied yet, it's cvs-tip Bugzilla as of last week just running on a backup copy of the production database from a week or two ago.

Spreadsheet tracking the process of porting the patches forward is at http://spreadsheets.google.com/ccc?key=p-AiBqJKo-QywsXTT5r2bQw

If you'd like to help port patches be my guest. :)  The patches are all public (urls are at the top of the spreadsheet).

Comment 13

[Frédéric Buclin]

Before upgrading to 3.0 RC1, you should fix all errors reported by sanitycheck.cgi. The list is *very* long, and you could get unexpected errors if you don't do it first.

Comment 14

[Dave Miller [:justdave]]

The times in comment 10 were based on running the upgrade scripts in a VM that was talking to a database running on a different VM.  Running it against a copy of bmo in a different directory on the production server talking to a backup copy of the database also on the production DB server was significantly faster.  The schema upgrade portion is 20 minutes shorter, and the charset conversion is 15 minutes shorter.  That said, I don't think I'll change the estimate. :)  Can always use the extra time for cleanup or troubleshooting if anything goes wrong.

What do we want to do about the charset conversion?  There's 716 fields in the database with an unguessable charset now.  Most of them are comments and summaries (and changes to summaries in the activity table).  I can get a list of the unmatched fields if anyone wants to try to decipher them (we can feed it per-item overrides if we find something wrong, and set a default charset to assume if it can't guess, which is probably windows-1252).

I'm thinking at this point that we just use the windows-1252 fallback and keep a backup of the database around to restore from if anyone finds anything strange.  It's going to be hell to try to manually figure out the several hundred unguessed entries.

Comment 15

[Dave Miller [:justdave]]

Oh, turns out there's an additional step to the charset conversion.  After the recoding is done, we have to go back and convert the table storage in the database so that MySQL actually knows that the data is UTF-8 (so sorting happens correctly and so forth).  That process just took 55 minutes to run.  So we're up 20 minutes now instead of down 35. :)

Comment 16

[Frédéric Buclin]

On the other hand, if some comments are encoded in an unknown charset, they are probably not displayed correctly in your browser and so there is not really dataloss here as we cannot display them in a readable manner anyway. I wouldn't worry too much about them.

Comment 17

[Dave Miller [:justdave]]

I forgot -- we get a little time back because we no longer have to upgrade Apache and perl modules and so forth.  All of the prereqs are in place on the production server already as of this last Tuesday.

So here's what our downtime schedule looks like now:

  15 min - pre-upgrade database backup
  60 min - 2.20 -> 3.0 schema conversion
  10 min - multiple charsets -> utf8 charset conversion
  60 min - iso-8859-1 -> utf8 MySQL storage conversion
  15 min - post-upgrade database backup
   0 (10) min - apache config for mod_perl (can be done during the 2nd backup)
--------
 160 min - total downtime if all goes well.
 (2h40m)

Built-in troubleshooting time:

 160 min - want to allow some time to attempt to fix minor things that might be
 (2h40m)   broken before resorting to restoring backups

In case of reversion:

  40 min - restore backups of database and bugzilla code (the newer apache and
           perl modules are generally well-tested upstream and should still run
           the older Bugzilla code, so attempting to revert those is not
           foreseen as potentially necessary)

========
 360 min (6 hours) - total downtime window

We've got a lot of recovery time built into the advertised outage window.  From how things are looking in testing it's probable that we won't need it.  So we're probably actually looking at about 3 hours for the full upgrade.

Comment 18

[Reed Loden [:reed]]

All done!

Comment 19

[Stephanie Daugherty [:sdaugherty]]

Yay.

Comment 20

[Dave Miller [:justdave]]

3.0rc1 isn't released yet, mostly as a result of things we found while staging this upgrade.  But we've upgraded anyway (shoved a LOT of fixes upstream in the last week).

Regressions are being tracked on bug 365063.