405404 - HTML::Scrubber throws "undef error - Wide character in subroutine entry" when a field filtered with html_light contains UTF-8 characters

Status: RESOLVED FIXED

(Bugzilla :: Bugzilla-General, defect)

Description

[A. Shimono [:himorin]]

bugzilla returns wide charactor error when show_bug
'Wide character in subroutine entry at /usr/lib/perl5/site_perl/5.8.5/HTML/Scrubber.pm line 322.'

bugzilla configurations are as following
# grep utf8 data/params 
           'utf8' => 1,

data are copied from current bugzilla.mozilla.gr.jp's (3.0+modified for -jp).
# so, you can see nearly the same one at http://bugzilla.mozilla.gr.jp/

* it seems error on TT when outputting mysql utf-8 flagged strings
* i had the same error on enter_bug.cgi
* could not fixed with binmode => ':utf8'
 my $output;
 $template->process("$format->{'template'}", $vars, \$output, binmode => ':utf8')
   || ThrowTemplateError($template->error());
 print $output;
* could use with utf8 = 0 in data/params
 garbaged on e-mail, but this might be another problem
* database was converted from 3.0(-ja) one (with checkconfig.pl)
* didn't try inserting '[% RAWPERL %]use utf8;[% END %]'

i have no idea to fix this currently, sorry.

Comment 1

[Frédéric Buclin]

Looks similar to bug 405362. Max?

Comment 2

[Frédéric Buclin]

OK, I can reproduce the bug. All you have to do is to inject UTF-8 characters (e.g. …) in a product/component/group description. As they are filtered using FILTER html_light, HTML::Scrubber is called and failed with:

undef error - Wide character in subroutine entry at /usr/lib/perl5/vendor_perl/5.8.8/HTML/Scrubber.pm line 322. Marc, as you reviewed the patch from bug 363153, could you investigate too?

Comment 3

[Marc Schumann [:Wurblzap]]

Hmm, my example of “Insídeṛ” is from a product description, which worked well... Strange... I'm investigating.

Comment 4

[Marc Schumann [:Wurblzap]]

I believe I must have used a product name after all, despite me thinking I checked a product description, too.

This starts working again if I force the code to use the part bracketed out by the

   if ($@ || $HTML::Parser::VERSION < 3.40) { # Package(s) not installed.

line. Does this mean it's a problem of certain versions of HTML::Parser?

Comment 5

[Frédéric Buclin]

No, this means we don't use HTML::Parser and HTML::Scrubber at all if you enter this part of the code. That's why you don't get the error anymore.

Comment 6

[Frédéric Buclin]

Attachment: patch, v1

Looks like HTML::Parser->utf8_mode(1) is no longer required. Note that I didn't test this patch on an installation with utf8 = 0.

man HTML::Parser:

$p->utf8_mode( $bool )
  Enable this option when parsing raw undecoded UTF-8. This tells the parser that the entities expanded for strings reported by "attr", @attr and "dtext" should be expanded as decoded UTF-8 so they end up compatible with the surrounding text.
  If "utf8_mode" is enabled then it is an error to pass strings containing characters with code above 255 to the parse() method, and the parse() method will croak if you try.

So if I understand the last sentence correctly, we shouldn't use utf8_mode anymore as we can now have wide characters.

Comment 7

[A. Shimono [:himorin]]

worked well with patch v1.

Comment 8

[Max Kanat-Alexander]

Comment on attachment 290215 [details] [diff] [review]
patch, v1

This looks OK. Do we still need to have HTML::Parser in the OPTIONAL_MODULES for checksetup?

Comment 9

[Frédéric Buclin]

Yes, as this module is still required to parse HTML tags in descriptions correctly.

Comment 10

[Max Kanat-Alexander]

(In reply to comment #9)
> Yes, as this module is still required to parse HTML tags in descriptions
> correctly.

  Okay. Is a specific *version* of it still required?

Comment 11

[Frédéric Buclin]

(In reply to comment #10)
>   Okay. Is a specific *version* of it still required?

I think we should still require 3.40 or better as we are sure it knows what to do with UTF8 stuff.

Comment 12

[Max Kanat-Alexander]

(In reply to comment #11)
> I think we should still require 3.40 or better as we are sure it knows what to
> do with UTF8 stuff.

  Okay, yes. The Changes for 3.39_90 support that decision.

Comment 13

[Frédéric Buclin]

Checking in Bugzilla/Util.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Util.pm,v  <--  Util.pm
new revision: 1.64; previous revision: 1.63
done

Comment 14

[Frédéric Buclin]

In the testcase, we should upgrade from a non-utf8 installation and see how fields using |FILTER html_light| are displayed. Be sure that utf8 = 0 in data/params (we already know the patch works fine when the param is turned on).