Upgrade from older Bugzilla broken by UTF-8

RESOLVED INVALID

Status

()

--
major
RESOLVED INVALID
10 years ago
10 years ago

People

(Reporter: ewilde, Unassigned)

Tracking

Details

(Reporter)

Description

10 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Build Identifier: Bugzilla 3.0.4

I upgraded from Bugzilla 2.22 to 3.0.4, following the instructions you
provided.  This process was by no means smooth.  Here's what I encountered and
how I had to fix it.

Upon launching the new Bugzilla:

Bugzilla has suffered an internal error. Please save this page and send it to
THE MAINTAINER HAS NOT YET BEEN SET with details of what you were doing at the
time this message appeared.

URL: http://jump-gate.homeworld:9180/bugzilla/

Template->process() failed twice.

First error: undef error - DBD::mysql::db selectrow_array failed: Illegal mix
  of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)
  for operation '=' [for Statement "SELECT subclass FROM setting WHERE
  name = ?"] at Bugzilla/User/Setting.pm line 102 Bugzilla::User::Setting::new(
  'Bugzilla::User::Setting','comment_sort_order',0,1,'oldest_to_newest',
  'oldest_to_newest',1) called at Bugzilla/User/Setting.pm line 214
Bugzilla::User::Setting::get_defaults() called at Bugzilla/User.pm line 344
Bugzilla::User::settings('Bugzilla::User=HASH(0x82eed04)') called at
  data/template/template/en/default/global/header.html.tmpl line 110
eval {...} called at data/template/template/en/default/global/header.html.tmpl
  line 110
eval {...} called at data/template/template/en/default/global/header.html.tmpl
  line 16
Template::Provider::__ANON__('Template::Context=HASH(0x8ca8588)') called at
  /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/Template/Document.pm
  line 155
eval {...} called at /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/
  Template/Document.pm line 153
Template::Document::process('Template::Document=HASH(0x8d2f880)',
  'Template::Context=HASH(0x8ca8588)') called at
  /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/Template/Context.pm
  line 350
eval {...} called at /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/
  Template/Context.pm line 320 Template::Context::process(
  'Template::Context=HASH(0x8ca8588)','global/header.html.tmpl',
  'HASH(0x8d13e10)') called at
   data/template/template/en/default/index.html.tmpl line 25
eval {...} called at data/template/template/en/default/index.html.tmpl line 16
Template::Provider::__ANON__('Template::Context=HASH(0x8ca8588)') called at
  /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/Template/Document.pm
  line 155
eval {...} called at /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/
  Template/Document.pm line 153
Template::Document::process('Template::Document=HASH(0x8cfe614)',
  'Template::Context=HASH(0x8ca8588)') called at
  /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/Template/Context.pm
  line 350
eval {...} called at /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/
  Template/Context.pm line 320
Template::Context::process('Template::Context=HASH(0x8ca8588)',
  'Template::Document=HASH(0x8cfe614)') called at
  /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/Template/Service.pm
  line 97
eval {...} called at /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/
  Template/Service.pm line 94 Template::Service::process(
  'Template::Service=HASH(0x8c7a0a4)','index.html.tmpl','HASH(0x836b870)')
  called at /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/Template.pm
  line 71
Template::process('Bugzilla::Template=HASH(0x8c34ffc)','index.html.tmpl',
  'HASH(0x836b870)') called at /var/www/SSEC/bugzilla/index.cgi line 71

Second error: undef error - DBD::mysql::db selectrow_array failed: Illegal mix
  of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE)
  for operation '=' [for Statement "SELECT subclass FROM setting WHERE
  name = ?"] at Bugzilla/User/Setting.pm line 102 Bugzilla::User::Setting::new(
  'Bugzilla::User::Setting','comment_sort_order',0,1,'oldest_to_newest',
  'oldest_to_newest',1) called at Bugzilla/User/Setting.pm line 214
And more of the same b.s.

You need to fix up checksetup.pl, which your documentation claims to know how
to fix all upgrades from earlier Bugzillas.  It doesn't.

First of all, it needs to do the following to the database:

  alter database Bugs character set utf8;
  alter database Bugs collate utf8_general_ci;

Then, it needs to do the following to each of the tables in the database (at
the very minimum, it needs to do the table shown):

  alter table setting convert to character set utf8;

I don't know what recode.pl is supposed to do but it appears useless.  Here's
its output:

  contrib/recode.pl --charset=cp1252
  Converting attachments.description...
  Converting attachments.mimetype...
  Converting attachments.filename...
  Converting bug_severity.value...
  Converting bug_status.value...
  Converting bugs.bug_file_loc...
  Converting bugs.bug_severity...
  Converting bugs.bug_status...
  Converting bugs.short_desc...
  Converting bugs.op_sys...
  Converting bugs.priority...
  Converting bugs.rep_platform...
  Converting bugs.version...
  Converting bugs.resolution...
  Converting bugs.target_milestone...
  Converting bugs.status_whiteboard...
  Converting bugs.keywords...
  Converting bugs.alias...
  Converting bugs_activity.added...
  Converting bugs_activity.removed...
  Converting classifications.name...
  Converting classifications.description...
  Converting components.name...
  Converting components.description...
  Converting fielddefs.name...
  Converting fielddefs.description...
  Converting flags.status...
  Converting flagtypes.name...
  Converting flagtypes.description...
  Converting flagtypes.cc_list...
  Converting flagtypes.target_type...
  Converting groups.name...
  Converting groups.description...
  Converting groups.userregexp...
  Converting keyworddefs.name...
  Converting keyworddefs.description...
  Converting logincookies.cookie...
  Converting logincookies.ipaddr...
  Converting longdescs.thetext...
  Converting longdescs.extra_data...
  Converting milestones.value...
  Converting namedqueries.name...
  Converting namedqueries.query...
  Converting op_sys.value...
  Converting priority.value...
  Converting products.name...
  Converting products.description...
  Converting products.milestoneurl...
  Converting products.defaultmilestone...
  Converting profile_setting.setting_name...
  Converting profile_setting.setting_value...
  Converting profiles.login_name...
  Converting profiles.cryptpassword...
  Converting profiles.realname...
  Converting profiles.disabledtext...
  Converting profiles.extern_id...
  Converting profiles_activity.oldvalue...
  Converting profiles_activity.newvalue...
  Converting quips.quip...
  Converting rep_platform.value...
  Converting resolution.value...
  Converting series.name...
  Converting series.query...
  Converting series_categories.name...
  Converting setting.name...
  Converting setting.default_value...
  Converting setting.subclass...
  Converting setting_value.name...
  Converting setting_value.value...
  Converting tokens.token...
  Converting tokens.tokentype...
  Converting tokens.eventdata...
  Converting versions.value...
  Converting whine_events.subject...
  Converting whine_events.body...
  Converting whine_queries.query_name...
  Converting whine_queries.title...
  Converting whine_schedules.run_day...
  Converting whine_schedules.run_time...

Even after doing this, I still got the error.  I also was not able to use the
--guess option because it wants some Perl module that I don't have installed.
There is absolutely no need to install any kind of Perl module, above the
regular database modules, to figure out what kind of character set the database
tables use.  Output from:

  show table status;

Shows the actual character set used for each table, no guessing required.

Incidentally, you could probably have avoided all of this by simply not forcing
UTF-8 on the read of the "setting" table and any other tables that are required
to do a login and get to the "Parameters" link in the page footer.  Then, the
user could have had a shot at turning UTF-8 off.  I know you think unicode and
being able to enter bugs in a foreign language is important but most of us
couldn't care less about it.  A dubious feature like that is especially
annoying when it prevents you from using what (until the upgrade was performed)
was a perfectly good, working system.

Or, here's another thought.  Maybe UTF-8 shouldn't be the default when you
upgrade from a pre-existing system.  As a matter of fact, your "Parameters"
page says as much:

  Existing databases should set this to true only after the data has been
  converted from existing legacy character encodings to UTF-8, using the
  contrib/recode.pl script.

Too bad you never get to see this page because the fact that its on precludes
you from getting there.  And, too bad recode.pl doesn't work.

Or, here's a third idea.  Maybe UTF-8 should be settable in localconfig (i.e.
something I can get to without requiring the assistance of the broken system).


Reproducible: Always

Steps to Reproduce:
1. See above
2.
3.
Actual Results:  
The upgrade didn't work until I fixed a whole bunch of stuff by hand.

Expected Results:  
Upgrades should go pretty seamlessly.  They certainly shouldn't take a working database and work it over so that its broken.

The steps to the upgrade are pretty clear.  The character set used should be left alone or the upgrade should optionally fix it (since some loss of data may occur).  And, really, there's no reason to force a particular character set at all, except for search ordering.

Comment 1

10 years ago
You didn't actually follow checksetup.pl's directions. You were supposed to run recode and then run checksetup.pl again. If you didn't do that, your upgrade is broken in various ways.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → INVALID
(Reporter)

Comment 2

10 years ago
Perhaps you miss the point.

Because of some arbitrary change that you made, upgrading from a prior, working version to a new version is now a total Goat Rope.

Instead of saying, "the user is an idiot," you might want to think about how you could make the upgrade process seamless.  That is, if you really want people to use your product (although I realize that it is freeware).

For example, you don't really need to enforce the new character set encoding that you chose.  You thought that UTF-8 was better for all of us but from the standpoint of typing in bug descriptions, Latin-1 is fine and dandy for the majority of us.

Or, you could make it possible so that the user could at least get to the point where they could turn it off, if they wanted to (like your documentation says can be done), either by moving the parameter to the outside configuration file or making startup more forgiving.

And, why do users have to run two programs?  From the their point of view, they are simply doing an upgrade.  It makes no nevermind to them that part of the process is recoding and part of the process is something else.  Don't you think the whole process is arduous enough without some artificial tests of instruction following ability?

And, while we're on the subject, why do the users have to run anything manually?  Seamless upgrade is usually a built-in part of most other installs.

Plus, there's the point of having to install needless Perl modules to make things work.  There's just no excuse for that.

So, ignore these remarks or not, its your prerogative but, from the ease of installation, makes me really want to keep using the product, point of view there are certainly a number of things that you could do to make things much better if you were so inclined.
(Reporter)

Comment 3

10 years ago
Oh, and by the way.  I did run recode.  It was still broken after I did that.  I think I said that in the original bug report.
You need to log in before you can comment on or make changes to this bug.