Last Comment Bug 431201 - recode.pl crash when trying to convert 2.16 database to 3.0.3 schema
: recode.pl crash when trying to convert 2.16 database to 3.0.3 schema
Status: RESOLVED FIXED
:
Product: Bugzilla
Classification: Server Software
Component: Installation & Upgrading (show other bugs)
: 3.0.3
: All Windows Server 2003
: -- major with 1 vote (vote)
: Bugzilla 3.2
Assigned To: Max Kanat-Alexander
: default-qa
Mentors:
: 462836 473141 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-04-28 12:09 PDT by Styve
Modified: 2009-02-20 13:55 PST (History)
9 users (show)
mkanat: approval+
mkanat: approval3.2+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
v1 (1.33 KB, patch)
2009-02-08 16:48 PST, Max Kanat-Alexander
LpSolit: review+
Details | Diff | Splinter Review

Description Styve 2008-04-28 12:09:44 PDT
User-Agent:       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; InfoPath.1)
Build Identifier: Bugzilla 3.0.3, ActiveStatePerl 5.8.8.822

When trying to recode an old database 2.16 to put it on my 3.0.3 fresh install. Recode.pl errors me with the following: 


C:\Bugzilla>contrib\recode.pl --charset=cp1252 
Converting attachments.description... 
Converting attachments.filename... 
Converting attachments.mimetype... 
Converting attachstatusdefs.name... 
Converting attachstatusdefs.description... 
Converting attachstatusdefs.product... 
Converting bug_severity.value... 
Converting bug_status.value... 
Converting bugs.clientcontact... 
Converting bugs.keywords... 
Converting bugs.short_desc... 
Wide character in subroutine entry at C:\Bugzilla\contrib\recode.pl 
line 251. 


Has recommanded by Max Kanat-Alexander <mkanat@bug...org> , my show table status:

mysql> SHOW TABLE STATUS;
+-----------------------------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+
----------+----------------------------------------+------------------------+
| Name                        | Engine | Version | Row_format | Rows  | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time         | Check_time          | Collation         |
 Checksum | Create_options                         | Comment                |
+-----------------------------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+
----------+----------------------------------------+------------------------+
| attach_data                 | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:01 | NULL                | NULL                | utf8_general_ci   |
     NULL | max_rows=100000 avg_row_length=1000000 | InnoDB free: 147456 kB |
| attachments                 | InnoDB |      10 | Compact    | 14898 |           9397 |   140001280 |               0 |       114688 |         0 |           1598 | 2008-04-24 13:42:12 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| attachstatusdefs            | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:42:26 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| attachstatuses              | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:42:26 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| bug_group_map               | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:01 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| bug_severity                | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        32768 |         0 |              1 | 2008-04-28 08:49:01 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| bug_status                  | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        32768 |         0 |              1 | 2008-04-28 08:49:01 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| bugs                        | InnoDB |      10 | Compact    |  3511 |            452 |     1589248 |               0 |      1294336 |         0 |           4241 | 2008-04-24 13:42:26 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| bugs_activity               | InnoDB |      10 | Compact    | 19747 |             80 |     1589248 |               0 |      1376256 |         0 |           NULL | 2008-04-24 13:42:32 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| bz_schema                   | InnoDB |      10 | Compact    |     1 |        1064960 |     1064960 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:01 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| category_group_map          | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| cc                          | InnoDB |      10 | Compact    |   621 |             79 |       49152 |               0 |        16384 |         0 |           NULL | 2008-04-24 13:42:53 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| classifications             | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |              1 | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| clients                     | InnoDB |      10 | Compact    |     2 |           8192 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| component_cc                | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| components                  | InnoDB |      10 | Compact    |   153 |            321 |       49152 |               0 |            0 |         0 |           NULL | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| dependencies                | InnoDB |      10 | Compact    |    94 |            174 |       16384 |               0 |        32768 |         0 |           NULL | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| duplicates                  | InnoDB |      10 | Compact    |   124 |            132 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| email_setting               | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| fielddefs                   | InnoDB |      10 | Compact    |    42 |            390 |       16384 |               0 |        32768 |         0 |             43 | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| flagexclusions              | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| flaginclusions              | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| flags                       | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        65536 |         0 |              1 | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| flagtypes                   | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |              1 | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| group_control_map           | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| group_group_map             | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| groups                      | InnoDB |      10 | Compact    |     7 |           2340 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| keyworddefs                 | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| keywords                    | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| logincookies                | InnoDB |      10 | Compact    |    12 |           1365 |       16384 |               0 |        16384 |         0 |            855 | 2008-04-24 13:42:54 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| longdescs                   | MyISAM |      10 | Dynamic    | 12305 |            250 |     3082060 | 281474976710655 |       408576 |         0 |           NULL | 2008-04-28 08:48:55 | 2008-04-28 08:48:55 | 2008-04-28 08:48:55 | latin1_swedish_ci |
     NULL |                                        |                        |
| milestones                  | InnoDB |      10 | Compact    |    19 |            862 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| namedqueries                | InnoDB |      10 | Compact    |   155 |            951 |      147456 |               0 |        16384 |         0 |           NULL | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| namedqueries_link_in_footer | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:02 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| namedquery_group_map        | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| op_sys                      | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        32768 |         0 |              1 | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| priority                    | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        32768 |         0 |              1 | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| products                    | InnoDB |      10 | Compact    |    19 |            862 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| profile_setting             | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| profiles                    | InnoDB |      10 | Compact    |    19 |           2586 |       49152 |               0 |        16384 |         0 |             34 | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| profiles_activity           | InnoDB |      10 | Compact    |    17 |            963 |       16384 |               0 |        49152 |         0 |           NULL | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| quips                       | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |              1 | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| rep_platform                | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        32768 |         0 |              1 | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| resolution                  | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        32768 |         0 |              1 | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| series                      | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |              1 | 2008-04-28 08:49:03 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| series_categories           | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |              1 | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| series_data                 | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| setting                     | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| setting_value               | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| shadowlog                   | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |              1 | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| status_workflow             | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| tokens                      | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| typebug                     | InnoDB |      10 | Compact    |    17 |            963 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| user_group_map              | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| versions                    | InnoDB |      10 | Compact    |   153 |            107 |       16384 |               0 |            0 |         0 |           NULL | 2008-04-24 13:43:05 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| votes                       | InnoDB |      10 | Compact    |     1 |          16384 |       16384 |               0 |        32768 |         0 |           NULL | 2008-04-24 13:43:06 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| watch                       | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |           NULL | 2008-04-24 13:43:06 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| whine_events                | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |            0 |         0 |              1 | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| whine_queries               | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        16384 |         0 |              1 | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
| whine_schedules             | InnoDB |      10 | Compact    |     0 |              0 |       16384 |               0 |        32768 |         0 |              1 | 2008-04-28 08:49:04 | NULL                | NULL                | latin1_swedish_ci |
     NULL |                                        | InnoDB free: 147456 kB |
+-----------------------------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+
----------+----------------------------------------+------------------------+
60 rows in set (0.05 sec)

mysql> 
Converting attachments.filename... 
Converting attachments.mimetype... 
Converting attachstatusdefs.name... 
Converting attachstatusdefs.description... 
Converting attachstatusdefs.product... 
Converting bug_severity.value... 
Converting bug_status.value... 
Converting bugs.clientcontact... 
Converting bugs.keywords... 
Converting bugs.short_desc... 
Wide character in subroutine entry at C:\Bugzilla\contrib\recode.pl 
line 251. 


C:\Bugzilla> 



Reproducible: Always

Steps to Reproduce:
1. Get a dump from an old 2.16 database with some french caracters (é è ê, ç, etc.)
2. Install a fresh 3.0.3 bugzilla on a Windows 3.0.3 server and get it running.
3. Drop your databse bugs, recreate one with your dump in step one.
4. Run checksetup.pl so it can modify and update the schema.
5. When ask, run recode.pl with the option --charset=cp1252
Actual Results:  
Can't log unto bugzilla with that database.

Expected Results:  
database conversion succes.

My old 2.16 was on a linux box. MySQL 3.23.54

My 3.0.3 is on a Windows 2003 server. MySQL 5.0.45-community-nt
Comment 1 Frédéric Buclin 2008-04-28 12:13:58 PDT
Max, can you confirm? If yes, I suppose that's a blocker.
Comment 2 Styve 2008-04-28 12:21:44 PDT
I can FTP you my sql dump. 130MEGS although.
Comment 3 Max Kanat-Alexander 2008-04-28 14:38:52 PDT
  I can look into it. FWIW, line 251 is:

  my $digest = md5_base64($data);

  So it's something with Digest::MD5, I guess.

  Styve, could you get me the output of:

  SHOW VARIABLES LIKE 'character_set_database'

  From your DB?
Comment 4 Styve 2008-04-28 15:02:45 PDT
Ok, I have restart the process to be sure to have clean date. Cause I have try
to UTF-8 Encode after the failure of recode.pl but with no success.

So, right now my db is up and I have run checksetup.pl one time so it has added
the new tables stucture and indexes. This works fine. My charset is

mysql> use bugs
Database changed
mysql> SHOW VARIABLES LIKE 'character_set_database'
    -> ;
+------------------------+--------+
| Variable_name          | Value  |
+------------------------+--------+
| character_set_database | latin1 |
+------------------------+--------+
1 row in set (0.00 sec)

Comment 5 Max Kanat-Alexander 2008-04-28 15:06:46 PDT
  This isn't critical and doesn't count as a "crash" (per the keyword)--it's part of the upgrading tools, not a part of Bugzilla itself, and isn't confirmed to affect any significant number of people. (It isn't even confirmed as a bug.)
Comment 6 Max Kanat-Alexander 2008-04-29 00:08:31 PDT
  Styve, I would like your dump, but I don't have an FTP site for you to upload it to, at the moment. If you want to put it up somewhere for me and email me directly with a link, that would work too.
Comment 7 Styve 2008-04-29 04:13:20 PDT
Ok, as soon as our network guy will be at the office, I'll send you the link, user and pwd to FTP the dump.
Comment 8 Kostomanolakis Stavros 2008-05-07 07:58:20 PDT
I can confirm this bug. I have the same problem with development release 3.1.4
My database is in uft8 format, and I use greek characters. The database has been migrated form a prior 3 bugzilla version.

...> contrib\recode.pl --guess --dry-run
Converting attachments.description...
Wide character in subroutine entry at C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\bugzilla\contrib\recode.pl line 252.

Working with this database and bugzilla version 4.1.3 works fine for me. The only serious problem that I have is that a "wide character in subroutine ..." internal error is caused when bugzilla tries to send email. I dont Know if this has something to do with the recode.pl problem

If it helps you, I can send my database dump with email (3.2 MB) 
Comment 9 Max Kanat-Alexander 2008-05-07 13:22:58 PDT
(In reply to comment #8)
> I can confirm this bug. I have the same problem with development release 3.1.4
> My database is in uft8 format, and I use greek characters. The database has
> been migrated form a prior 3 bugzilla version.

  If your database is already utf8-encoded, you don't need to recode it.
Comment 10 Heiko Seitz 2008-06-03 07:39:12 PDT
Hm, I'm having the same problem at the moment. I'm trying to switch to an Oracle database. Unfortunately I'll have to merge two existing Bugzillas into one. 

The newer one is running with 2.22 with UTF8 switched to on in the admin pages since the beginning. No issues here with upgrading to 3.1.4 of Bugzilla. 

The older one (bug number 1 dates 2001) is running with 3.0 at the moment (sounds stupid but this is indeed the older installation) and was upgraded several times over the years. I cannot figure out how often, what versions or even what he first version was to begin with since I've taken over this job from someone who left the company a while ago. This old installation is running with UTF8 deactivated in the admin pages and probably does so since creation. This old database is much larger than the newer one.

However, I tried to run recode.pl like:
./contrib/recode.pl --dry-run --guess --show-failures --charset=cp1252

Omitting the --guess or the --dry-run doesn't change the result. The conversions starts and apparently succeeds with some columns but suddenly aborts with:
"Wide character in subroutine entry at ./contrib/recode.pl line 251."

My SHOW VARIABLES LIKE 'character_set_database' matches that of Styve. Collation is latin1_swedish_ci. Both databases (the working and the not working) contain german characters like ä, ö, ü or ß.
Comment 11 David Sawicki 2008-10-29 07:02:12 PDT
I'm experiencing the same problem in the same location of the script... trying to convert a database from 2.20.1 to 3.2rc1 using the 

recode.pl --charset=latin1

I considered opening a new bug, only because the OS, perl version, and mysql version wouldn't match.  I'm using Fedora 9 Linux, perl version 5.10.0, and mysql 5.0.51 which both come default with Fedora 9.  

the dumpfile I'm using is 1.6 GB in size.  Any suggestions to try... I'm using a test data set and have the ability to reload the database... but it is a copy of the real data.  I have the flexibility to use 'test' script if needed to verify that the data converted properly.

Thx in advance.  Regards,

Dave
Comment 12 Max Kanat-Alexander 2008-11-03 03:35:30 PST
*** Bug 462836 has been marked as a duplicate of this bug. ***
Comment 13 Max Kanat-Alexander 2008-11-03 03:49:14 PST
*** Bug 462836 has been marked as a duplicate of this bug. ***
Comment 14 Mike 2009-01-02 08:52:39 PST
I'm also seeing this in the bugzilla 3.2 install that I'm attempting.

My database dump was from a 2.20 install. The following is my output:
# contrib/recode.pl --dry-run --guess
Converting attachments.description...
Converting attachments.filename...
Converting attachments.mimetype...
Converting bug_severity.value...
Converting bug_status.value...
Converting bugs.priority...
Converting bugs.keywords...
Converting bugs.bug_file_loc...
Converting bugs.rep_platform...
Converting bugs.short_desc...
Wide character in subroutine entry at contrib/recode.pl line 251.

Also:+------------------------+--------+
| Variable_name          | Value  |
+------------------------+--------+
| character_set_database | latin1 | 
+------------------------+--------+
1 row in set (0.00 sec)

This is on an Ubuntu installation, using mysql 5.0.
Comment 15 Albert Ting 2009-01-02 15:36:55 PST
(In reply to comment #14)
> If your database is already utf8-encoded, you don't need to recode it.

My database is indeed already utf8 encoded.  The problem is that when upgrading to 3.2, checksetup will create a new table, bugs_fulltext in non-utf8 mode.  This makes checksetup think the entire database is not in utf8 and outputs the warning to run recode.pl, which as you noted isn't necessary. 

This doesn't solve the "wide character" problem, but it helps alleviate it for most folks.  That is, it would be nice if checksetup automatically created new tables in UTF8 mode to begin with, thereby avoiding the warning message altogether.
Comment 16 Max Kanat-Alexander 2009-01-02 15:43:39 PST
(In reply to comment #15)
> My database is indeed already utf8 encoded.  The problem is that when upgrading
> to 3.2, checksetup will create a new table, bugs_fulltext in non-utf8 mode. 

  Oh, because you were coming from 2.20 and so the utf8 parameter is off? I think we can handle that situation! We just have to enable the utf8 parameter if the DB is already utf8.

> It would be nice if checksetup automatically created new
> tables in UTF8 mode to begin with, thereby avoiding the warning message
> altogether.

  It does, as long as you have the utf8 parameter enabled. But of course, upgrading from a version that didn't have the parameter at all, it would be disabled by default even if your DB is utf8 (which is a fairly rare case, which is why I suppose we didn't catch it during testing).
Comment 17 Albert Ting 2009-01-02 17:09:28 PST
(In reply to comment #16)
> (In reply to comment #15)
> > My database is indeed already utf8 encoded.  The problem is that when upgrading
> > to 3.2, checksetup will create a new table, bugs_fulltext in non-utf8 mode. 
> 
>   Oh, because you were coming from 2.20 and so the utf8 parameter is off? I
> think we can handle that situation! We just have to enable the utf8 parameter
> if the DB is already utf8.

Nope, I already have utf8=1, as I was upgrading from 3.1.2 to 3.2.

> > It would be nice if checksetup automatically created new
> > tables in UTF8 mode to begin with, thereby avoiding the warning message
> > altogether.
> 
>   It does, as long as you have the utf8 parameter enabled. But of course,
> upgrading from a version that didn't have the parameter at all, it would be
> disabled by default even if your DB is utf8 (which is a fairly rare case, which is why I suppose we didn't catch it during testing).

It could be my environment, but utf8 was already set before I ran checksetup.
Comment 18 Albert Ting 2009-01-02 17:14:17 PST
In Bugzilla/DB/Mysql.pm, I see this code:

    my $utf_table_status =
        $self->selectall_arrayref("SHOW TABLE STATUS", {Slice=>{}});
    $self->_after_table_status([map($_->{Name}, @$utf_table_status)]);
    my @non_utf8_tables = grep($_->{Collation} !~ /^utf8/, @$utf_table_status);
    
    if (Bugzilla->params->{'utf8'} && scalar @non_utf8_tables) {
        print <<EOT;
        WARNING: We are about to convert....

Since the new bugs_fulltext table was created in non-utf8 mode (and I was already in UTF8 mode), checksetup outputs the above warning message.  

I think the solution should have the "CREATE TABLE" code automatically create new tables in UTF8 mode.
Comment 19 Max Kanat-Alexander 2009-01-02 17:18:59 PST
(In reply to comment #18)
> I think the solution should have the "CREATE TABLE" code automatically create
> new tables in UTF8 mode.

  If you read the code, you'll see that it already does. You must have changed the global charset of your database in order for this to occur on your installation, but we're supposed to handle that case also...
Comment 20 Albert Ting 2009-01-02 17:30:54 PST
(In reply to comment #19)
> (In reply to comment #18)
>   If you read the code, you'll see that it already does. You must have changed
> the global charset of your database in order for this to occur on your
> installation, but we're supposed to handle that case also...

Could you let me know where? Schema.pm's get_create_database_sql and get_create_table_ddl doesn't add any extra stuff. I do see get_table_ddl append DB_EXTRAS, but I don't see that defined anywhere, including bz_schema.
Comment 21 Max Kanat-Alexander 2009-01-02 18:05:51 PST
(In reply to comment #20)
> Could you let me know where?

  Bugzilla::DB::Schema::Mysql
Comment 22 Albert Ting 2009-01-02 18:48:00 PST
Ah, figured out where I went wrong.  I'm rebuilding the database from a backup, so I must now add a 'character set utf8" to my "create database" call.  This solved my issue.  So false alarm in my situation.
Comment 23 Frédéric Buclin 2009-01-12 12:34:49 PST
*** Bug 473141 has been marked as a duplicate of this bug. ***
Comment 24 Max Kanat-Alexander 2009-02-08 16:48:40 PST
Created attachment 361197 [details] [diff] [review]
v1

This patch should fix it. Anybody who is having this problem: Could you try this out for me and tell me if it fixes things?

I also fixed the instructions for installing Encode::Detect, since I was here anyway and it's a contrib script.
Comment 25 Frédéric Buclin 2009-02-08 16:53:04 PST
Comment on attachment 361197 [details] [diff] [review]
v1

I didn't test it, but it looks good. I will let someone with a "broken" DB test it. r=LpSolit

Max, is it intentional that you don't check the value of the utf8 parameter?
Comment 26 Max Kanat-Alexander 2009-02-08 16:54:51 PST
(In reply to comment #25)
> Max, is it intentional that you don't check the value of the utf8 parameter?

  It is--md5_hex *always* dies here w/ utf8 characters, and people can theoretically run this script against a DB w/o turning on the utf8 param.
Comment 27 Max Kanat-Alexander 2009-02-08 16:55:32 PST
Putting this into the approval queue pending verification that it fixes the problem for people. (Although 3.0 should never have been affected by the bug that this fixes...)
Comment 28 Frédéric Buclin 2009-02-20 06:17:10 PST
Max, I suggest we commit this patch. Nobody took the time to reply in two weeks.
Comment 29 Max Kanat-Alexander 2009-02-20 13:52:53 PST
Okay, will do.
Comment 30 Max Kanat-Alexander 2009-02-20 13:55:39 PST
tip:

Checking in contrib/recode.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/recode.pl,v  <--  recode.pl
new revision: 1.6; previous revision: 1.5
done

3.2:

Checking in contrib/recode.pl;
/cvsroot/mozilla/webtools/bugzilla/contrib/recode.pl,v  <--  recode.pl
new revision: 1.5.2.1; previous revision: 1.5
done

Note You need to log in before you can comment on or make changes to this bug.