Last Comment Bug 771100 - Unable to attach a file to a bug with perl 5.16.0
: Unable to attach a file to a bug with perl 5.16.0
Status: RESOLVED FIXED
:
Product: Bugzilla
Classification: Server Software
Component: Attachments & Requests (show other bugs)
: 4.2.1
: All All
: -- major (vote)
: Bugzilla 4.0
Assigned To: Frédéric Buclin
: default-qa
Mentors:
https://rt.perl.org/rt3/Public/Bug/Di...
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-05 05:11 PDT by Per Pomsel
Modified: 2013-01-21 04:30 PST (History)
6 users (show)
LpSolit: approval+
LpSolit: blocking4.4+
LpSolit: approval4.2+
LpSolit: blocking4.2.3+
LpSolit: approval4.0+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
PoC (903 bytes, text/plain)
2012-08-14 15:17 PDT, Frédéric Buclin
no flags Details
PoC (917 bytes, text/plain)
2012-08-14 15:37 PDT, Frédéric Buclin
no flags Details
patch, v1 (442 bytes, patch)
2012-08-14 17:32 PDT, Frédéric Buclin
dkl: review+
Details | Diff | Splinter Review

Description Per Pomsel 2012-07-05 05:11:21 PDT
Hello,

after updating perl to version 5.16.0 I'm unable to attach files to a bug ("You did not specify a file to attach").
With perl 5.12.4 it's working fine.
Results from checksetup.pl:

* This is Bugzilla 4.2.1 on perl 5.16.0
* Running on Linux 3.4.4-gentoo #1 SMP Sun Jun 24 09:06:32 CEST 2012

Checking perl modules...
Checking for               CGI.pm (v3.51)     ok: found v3.59 
Checking for           Digest-SHA (any)       ok: found v5.71 
Checking for             TimeDate (v2.21)     ok: found v2.24 
Checking for             DateTime (v0.28)     ok: found v0.76 
Checking for    DateTime-TimeZone (v0.71)     ok: found v1.46 
Checking for                  DBI (v1.614)    ok: found v1.622 
Checking for     Template-Toolkit (v2.22)     ok: found v2.24 
Checking for           Email-Send (v2.00)     ok: found v2.198 
Checking for           Email-MIME (v1.904)    ok: found v1.910 
Checking for                  URI (v1.37)     ok: found v1.60 
Checking for       List-MoreUtils (v0.22)     ok: found v0.33 
Checking for    Math-Random-ISAAC (v1.0.1)    ok: found v1.004 

Checking available perl DBD modules...
Checking for               DBD-Pg (v1.45)     ok: found v2.19.2 
Checking for            DBD-mysql (v4.001)    not found 
Checking for           DBD-SQLite (v1.29)     not found 
Checking for           DBD-Oracle (v1.19)     not found 

The following Perl modules are optional:
Checking for                   GD (v1.20)     ok: found v2.46 
defined(@array) is deprecated at /usr/lib/perl5/vendor_perl/5.16.0/Chart/Base.pm line 181.
        (Maybe you should just omit the defined()?)
defined(@array) is deprecated at /usr/lib/perl5/vendor_perl/5.16.0/Chart/Base.pm line 233.
        (Maybe you should just omit the defined()?)
Checking for                Chart (v2.1)      ok: found v2.4.5 
Checking for          Template-GD (any)       ok: found v1.56 
Checking for           GDTextUtil (any)       ok: found v0.86 
Checking for              GDGraph (any)       ok: found v1.44 
Checking for           MIME-tools (v5.406)    ok: found v5.503 
Checking for          libwww-perl (any)       ok: found v6.04 
$[ used in numeric lt (<) (did you mean $] ?) at /usr/lib/perl5/vendor_perl/5.16.0/XML/Twig.pm line 7286.
$[ used in numeric lt (<) (did you mean $] ?) at /usr/lib/perl5/vendor_perl/5.16.0/XML/Twig.pm line 7292.
$[ used in numeric lt (<) (did you mean $] ?) at /usr/lib/perl5/vendor_perl/5.16.0/XML/Twig.pm line 7304.
Checking for             XML-Twig (any)       ok: found v3.39 
Checking for          PatchReader (v0.9.6)    ok: found v0.9.6 
Checking for            perl-ldap (any)       ok: found v0.44 
Checking for          Authen-SASL (any)       ok: found v2.15 
Checking for           RadiusPerl (any)       not found 
Checking for            SOAP-Lite (v0.712)    ok: found v0.714 
Checking for             JSON-RPC (any)       ok: found v1.03 
Checking for              JSON-XS (v2.0)      ok: found v2.32 
Checking for           Test-Taint (any)       ok: found v1.04 
Checking for          HTML-Parser (v3.67)     ok: found v3.69 
Checking for        HTML-Scrubber (any)       ok: found v0.09 
Checking for               Encode (v2.21)     ok: found v2.44 
Checking for        Encode-Detect (any)       ok: found v1.01 
Checking for Email-MIME-Attachment-Stripper (any)       ok: found v1.316 
Checking for          Email-Reply (any)       ok: found v1.202 
Checking for          TheSchwartz (any)       ok: found v1.10 
Checking for       Daemon-Generic (any)       ok: found v0.82 
Checking for             mod_perl (v1.999022) not found 
Checking for     Apache-SizeLimit (v0.96)     not found 
Checking for          mod_headers (any)       ok 
Checking for          mod_expires (any)       ok 
Checking for              mod_env (any)       ok 
***********************************************************************
* OPTIONAL MODULES                                                    *
***********************************************************************
* Certain Perl modules are not required by Bugzilla, but by           *
* installing the latest version you gain access to additional         *
* features.                                                           *
*                                                                     *
* The optional modules you do not have installed are listed below,    *
* with the name of the feature they enable. Below that table are the  *
* commands to install each module.                                    *
***********************************************************************
*      MODULE NAME * ENABLES FEATURE(S)                               *
***********************************************************************
*       RadiusPerl * RADIUS Authentication                            *
*         mod_perl * mod_perl                                         *
* Apache-SizeLimit * mod_perl                                         *
***********************************************************************
COMMANDS TO INSTALL OPTIONAL MODULES:

     RadiusPerl: /usr/bin/perl5.16.0 install-module.pl Authen::Radius
       mod_perl: /usr/bin/perl5.16.0 install-module.pl mod_perl2
Apache-SizeLimit: /usr/bin/perl5.16.0 install-module.pl Apache2::SizeLimit


To attempt an automatic install of every required and optional module
with one command, do:

  /usr/bin/perl5.16.0 install-module.pl --all

Reading ./localconfig...

OPTIONAL NOTE: If you want to be able to use the 'difference between two
patches' feature of Bugzilla (which requires the PatchReader Perl module
as well), you should install patchutils from:

    http://cyberelk.net/tim/patchutils/

Checking for               DBD-Pg (v1.45)     ok: found v2.19.2 
Checking for           PostgreSQL (v8.03.0000) ok: found v09.01.0400 
Checking for               DBD-Pg (v2.17.2)   ok: found v2.19.2 

Removing existing compiled templates...
Precompiling templates...done.
Fixing file permissions...
Deriving regex group memberships...
checksetup.pl complete.
Comment 1 john.manko 2012-08-13 11:54:33 PDT
I'm receiving the error.  Here is my system:

Gentoo 3.4.4-gentoo
Perl v5.16.1
Apache v2.2.22-r1
mod_perl v2.0.7

modules:
dev-perl/Any-Moose-0.180.0:0
dev-perl/AnyEvent-7.10.0:0
dev-perl/Apache-DBI-1.110.0:0
dev-perl/Apache-Reload-0.120.0:0
dev-perl/Apache-SizeLimit-0.960.0:0
dev-perl/Apache-Test-1.370.0:0
dev-perl/AppConfig-1.660.0:0
dev-perl/Authen-SASL-2.150.0:0
dev-perl/B-Hooks-EndOfScope-0.110.0:0
dev-perl/Capture-Tiny-0.180.0:0
dev-perl/Chart-2.4.5:0
dev-perl/Class-Accessor-0.340.0:0
dev-perl/Class-Accessor-Lite-0.50.0:0
dev-perl/Class-C3-0.230.0:0
dev-perl/Class-Data-Inheritable-0.80.0:0
dev-perl/Class-Inspector-1.270.0:0
dev-perl/Class-Load-0.200.0:0
dev-perl/Class-Load-XS-0.40.0:0
dev-perl/Class-Singleton-1.400.0:0
dev-perl/Class-Trigger-0.140.0:0
dev-perl/Convert-ASN1-0.260.0:0
dev-perl/Crypt-Random-Source-0.70.0:0
dev-perl/Crypt-SSLeay-0.580.0:0
dev-perl/DBD-Pg-2.19.2:0
dev-perl/DBD-mysql-4.20.0:0
dev-perl/DBI-1.622.0:0
dev-perl/Daemon-Generic-0.820.0:0
dev-perl/Data-ObjectDriver-0.90.0:0
dev-perl/Data-OptList-0.107.0:0
dev-perl/DateTime-0.760.0:0
dev-perl/DateTime-Format-DateParse-0.50.0:0
dev-perl/DateTime-Format-Mail-0.300.100:0
dev-perl/DateTime-Format-W3CDTF-0.60.0:0
dev-perl/DateTime-Locale-0.450.0:0
dev-perl/DateTime-TimeZone-1.480.0:0
dev-perl/Devel-GlobalDestruction-0.90.0:0
dev-perl/Devel-StackTrace-1.270.0:0
dev-perl/Devel-StackTrace-AsHTML-0.110.0:0
dev-perl/Digest-HMAC-1.30.0:0
dev-perl/Digest-SHA1-2.130.0:0
dev-perl/Dist-CheckConflicts-0.20.0:0
dev-perl/Email-Abstract-3.3.0:0
dev-perl/Email-Address-1.896.0:0
dev-perl/Email-Date-Format-1.2.0:0
dev-perl/Email-MIME-1.911.0:0
dev-perl/Email-MIME-Attachment-Stripper-1.316.0:0
dev-perl/Email-MIME-ContentType-1.15.0:0
dev-perl/Email-MIME-Encodings-1.313.0:0
dev-perl/Email-MessageID-1.402.0:0
dev-perl/Email-Reply-1.202.0:0
dev-perl/Email-Send-2.198.0:0
dev-perl/Email-Simple-2.102.0:0
dev-perl/Encode-Detect-1.10.0:0
dev-perl/Encode-Locale-1.30.0:0
dev-perl/Error-0.170.180:0
dev-perl/Eval-Closure-0.80.0:0
dev-perl/File-Flock-2008.10.0:0
dev-perl/File-HomeDir-0.990.0:0
dev-perl/File-Listing-6.40.0:0
dev-perl/File-ShareDir-1.30.0:0
dev-perl/File-Slurp-9999.190.0:0
dev-perl/File-Which-1.90.0:0
dev-perl/Filesys-Notify-Simple-0.80.0:0
dev-perl/GD-2.460.0:0
dev-perl/GD-Graph3d-0.630.0:0
dev-perl/GDGraph-1.440.0:0
dev-perl/GDTextUtil-0.860.0:0
dev-perl/HTML-Parser-3.690.0:0
dev-perl/HTML-Scrubber-0.90.0:0
dev-perl/HTML-TableParser-0.380.0:0
dev-perl/HTML-Tagset-3.200.0:0
dev-perl/HTTP-Body-1.150.0:0
dev-perl/HTTP-Cookies-6.0.1:0
dev-perl/HTTP-Daemon-6.10.0:0
dev-perl/HTTP-Date-6.20.0:0
dev-perl/HTTP-Message-6.30.0:0
dev-perl/HTTP-Negotiate-6.0.1:0
dev-perl/Hash-MultiValue-0.120.0:0
dev-perl/IO-Socket-SSL-1.760.0:0
dev-perl/IO-stringy-2.110.0:0
dev-perl/JSON-2.530.0:0
dev-perl/JSON-RPC-1.30.0:0
dev-perl/JSON-XS-2.330.0:0
dev-perl/LWP-MediaTypes-6.20.0:0
dev-perl/LWP-Protocol-https-6.30.0:0
dev-perl/Linux-Pid-0.40.0:0
dev-perl/List-MoreUtils-0.330.0:0
dev-perl/Locale-gettext-1.50.0:0
dev-perl/MIME-Lite-3.28.0:0
dev-perl/MIME-Types-1.350.0:0
dev-perl/MIME-tools-5.503.0:0
dev-perl/MRO-Compat-0.110.0:0
dev-perl/MailTools-2.90.0:0
dev-perl/Math-Random-ISAAC-1.4.0:0
dev-perl/Math-Random-ISAAC-XS-1.4.0:0
dev-perl/Math-Random-Secure-0.60.0:0
dev-perl/Math-Round-0.60.0:0
dev-perl/Module-Find-0.110.0:0
dev-perl/Module-Implementation-0.60.0:0
dev-perl/Module-Runtime-0.13.0:0
dev-perl/Moose-2.60.300:0
dev-perl/Net-Daemon-0.480.0:0
dev-perl/Net-HTTP-6.30.0:0
dev-perl/Net-SMTP-SSL-1.10.0:0
dev-perl/Net-SSLeay-1.480.0-r1:0
dev-perl/Package-DeprecationManager-0.130.0:0
dev-perl/Package-Stash-0.330.0:0
dev-perl/Package-Stash-XS-0.250.0:0
dev-perl/Params-Util-1.60.0:0
dev-perl/Params-Validate-1.60.0:0
dev-perl/PatchReader-0.9.6:0
dev-perl/PlRPC-0.202.0:0
dev-perl/Plack-1.0.100:0
dev-perl/Return-Value-1.666.1:0
dev-perl/Router-Simple-0.90.0:0
dev-perl/SOAP-Lite-0.715.0:0
dev-perl/Sub-Exporter-0.984.0:0
dev-perl/Sub-Exporter-Progressive-0.1.4:0
dev-perl/Sub-Identify-0.40.0:0
dev-perl/Sub-Install-0.926.0:0
dev-perl/Sub-Name-0.50.0:0
dev-perl/Template-DBI-2.650.0:0
dev-perl/Template-GD-2.660.0:0
dev-perl/Template-Latex-2.170.0:0
dev-perl/Template-Toolkit-2.240.0:0
dev-perl/Template-XML-2.170.0:0
dev-perl/TermReadKey-2.300.0:0
dev-perl/Test-SharedFork-0.200.0:0
dev-perl/Test-TCP-1.160.0:0
dev-perl/Test-Taint-1.40.0:0
dev-perl/Text-Iconv-1.700.0:0
dev-perl/TheSchwartz-1.100.0:0
dev-perl/Tie-IxHash-1.220.0:0
dev-perl/TimeDate-1.200.0:0
dev-perl/Try-Tiny-0.110.0:0
dev-perl/URI-1.600.0:0
dev-perl/Variable-Magic-0.500.0:0
dev-perl/WWW-RobotRules-6.10.0:0
dev-perl/XML-DOM-1.440.0:0
dev-perl/XML-Filter-BufferText-1.10.0:0
dev-perl/XML-Handler-YAWriter-0.230.0:0
dev-perl/XML-LibXML-2.0.300:0
dev-perl/XML-NamespaceSupport-1.110.0:0
dev-perl/XML-Parser-2.410.0-r1:0
dev-perl/XML-RSS-1.490.0:0
dev-perl/XML-RegExp-0.40.0:0
dev-perl/XML-SAX-0.990.0:0
dev-perl/XML-SAX-Base-1.80.0:0
dev-perl/XML-SAX-Writer-0.530.0:0
dev-perl/XML-Simple-2.200.0:0
dev-perl/XML-Twig-3.390.0:0
dev-perl/XML-XPath-1.130.0:0
dev-perl/common-sense-3.600.0:0
dev-perl/libwww-perl-6.40.0:0
dev-perl/libxml-perl-0.80.0:0
dev-perl/namespace-clean-0.230.0:0
dev-perl/perl-ldap-0.440.0:0
dev-perl/text-autoformat-1.669.2:0
dev-perl/text-reform-1.200.0:0
Comment 2 Frédéric Buclin 2012-08-13 13:58:53 PDT
Confirmed. I get the same error message with Mageia 3 and Perl 5.16.0.
Comment 3 Frédéric Buclin 2012-08-13 14:28:30 PDT
Using CGI 3.59 on Perl 5.14.2 works fine.
Using CGI 3.59 on Perl 5.16.0 fails.

$cgi->param('data') correctly shows the name of the uploaded file, but $cgi->upload('data') is undefined, making Bugzilla to think that the file is missing.

Mark: are you aware of this bug?
Comment 4 Frédéric Buclin 2012-08-13 15:38:04 PDT
It looks like the /usr/tmp/CGItempxxx file is not created at all. If I run this code:

    my ($k) = keys %{$cgi->{'.tmpfiles'}};
    my $n = $cgi->{'.tmpfiles'}->{$k}->{name};
    my $f = Dumper($n);
    $f =~ m{/usr/tmp/CGItemp(\d+)};
    my $i = $1;
    open(FH, "<", "/usr/tmp/CGItemp$i") or die $!;
    my $s = -s FH;
    die $s;

open() is unable to find the file. So nothing is uploaded.
Comment 5 Alexei Volkov 2012-08-14 12:24:06 PDT
Have the same issue on my FreeBSD 9.0-RELEASE-p3 / Apache 2.2.22 / mod_perl 2.0.7 / Perl v5.16.0 / Bugzilla 4.2.2 right after upgrading to perl 5.16.

It seems that the real issue is about CGI.pm specifics to perl 5.16.

I have no fix or workaround so far.
Comment 6 Frédéric Buclin 2012-08-14 14:54:15 PDT
I'm not so sure it's a bug in CGI.pm itself, or at least it's more subtle than that. If I use CGI directly instead of Bugzilla::CGI, then I can upload files without any problem. So something is maybe wrong on our side.
Comment 7 Frédéric Buclin 2012-08-14 15:17:03 PDT
Created attachment 651907 [details]
PoC

Here is the script I used for my testing. The single line you need to edit to test with Bugzilla::CGI or CGI alone is line 12:

my $cgi = $bz_cgi;  # Use Bugzilla::CGI
my $cgi = $cpan_cgi # Use CGI only

With Perl 5.16, uploading files work fine with $cpan_cgi, but fails with $bz_cgi.
Comment 8 Frédéric Buclin 2012-08-14 15:37:52 PDT
Created attachment 651916 [details]
PoC

This PoC removes uninitialized warnings.
Comment 9 Frédéric Buclin 2012-08-14 15:49:38 PDT
OK, I found what's wrong. The problem comes from Bugzilla::CGI::_fix_utf8(). For some reason which I don't understand yet, this line is causing trouble in Perl 5.16 only:

    # The is_utf8 is here in case CGI gets smart about utf8 someday.
    utf8::decode($input) if defined $input && !utf8::is_utf8($input);

If I remove this line, then everything works fine.
Comment 10 Frédéric Buclin 2012-08-14 16:31:56 PDT
The problem comes from utf8::decode($input). I printed the content of Dumper($input) right before and right after calling utf8::decode($input), and the result is different in Perl 5.14 and 5.16.

5.14.2:

before: $VAR1 = bless( \*{'Fh::fh00001bug0-6.diff'}, 'Fh' );
after:  $VAR1 = bless( \*{'Fh::fh00001bug0-6.diff'}, 'Fh' );

5.16.0:

before: $VAR1 = bless( \*{'Fh::fh00001bug0-6.diff'}, 'Fh' );
after:  $VAR1 = 'bug0-6.diff'

The results above are the same for $input = $cgi->param('file') and
$input = $cgi->upload('file').

I don't know if this change in Perl 5.16 is intentional or not, but it looks like we can fix the problem ourselves. Patch coming!
Comment 11 Frédéric Buclin 2012-08-14 16:36:49 PDT
(In reply to Frédéric Buclin from comment #10)
> after:  $VAR1 = '<missing bug number>-6.diff'

Arhh, bmo mangled what I wrote, because it thought I was talking about bug zero. Let's say it should be:

 after:  $VAR1 = 'bugO-6.diff'
Comment 12 Frédéric Buclin 2012-08-14 17:25:28 PDT
Per https://rt.perl.org/rt3/Public/Bug/Display.html?id=91852, this behavior change is intentional and has been fixed in Perl 5.16. It was considered a bug.

I'm removing Mark from the CC list as his module is not the culprit and I don't want to spam him any longer. :)
Comment 13 Frédéric Buclin 2012-08-14 17:32:08 PDT
Created attachment 651946 [details] [diff] [review]
patch, v1

If it's a filehandle, then we must not decode it anymore as it will be automatically stringified now.
Comment 14 Alexei Volkov 2012-08-14 21:29:24 PDT
Great thanks to Frédéric Buclin , attachment 651946 [details] [diff] [review] works for me as well.
Comment 15 john.manko 2012-08-15 11:50:56 PDT
I confirm that the patch worked for me, too.
Comment 16 David Lawrence [:dkl] 2012-08-15 12:12:52 PDT
Comment on attachment 651946 [details] [diff] [review]
patch, v1

Review of attachment 651946 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl
Comment 17 Frédéric Buclin 2012-08-15 12:31:03 PDT
Committing to: bzr+ssh://lpsolit%40gmail.com@bzr.mozilla.org/bugzilla/trunk/
modified Bugzilla/CGI.pm
Committed revision 8346.

Committing to: bzr+ssh://lpsolit%40gmail.com@bzr.mozilla.org/bugzilla/4.2/
modified Bugzilla/CGI.pm
Committed revision 8119.
Comment 18 Alexei Volkov 2012-08-16 22:07:13 PDT
I was happy with that solution, but not so long. In fact the files attached with the most recent patch are broken. For the instance, attaching any zip archive via web form you can download it back, but it can't be unzipped anymore.
Comment 19 Frédéric Buclin 2012-08-17 05:19:03 PDT
(In reply to Alexei Volkov from comment #18)
> with the most recent patch are broken. For the instance, attaching any zip
> archive via web form you can download it back, but it can't be unzipped
> anymore.

I cannot reproduce your issue. Binary files are correctly uploaded/downloaded, and I can open them as expected. My patch has nothing to do with the content of attachments anyway, it only prevented the filehandle from being converted into a string.
Comment 20 Byron Jones ‹:glob› 2013-01-20 21:39:21 PST
this patch also addresses issues with searching multi-value fields on 4.0.

given this patch is trivial, breaks core bugzilla functionality, and is being encountered more frequently as distros upgrade perl to 5.16, requesting approval for a 4.0 commit (to ride along with the next security release).
Comment 21 Frédéric Buclin 2013-01-21 04:27:19 PST
(In reply to Byron Jones ‹:glob› from comment #20)
> being encountered more frequently as distros upgrade perl to 5.16,

I would expect that if distros provide Perl 5.16, they also provide a pretty recent version of Bugzilla, i.e. 4.2. But it doesn't hurt to take it for 4.0 too.
Comment 22 Frédéric Buclin 2013-01-21 04:30:19 PST
Committing to: bzr+ssh://lpsolit%40gmail.com@bzr.mozilla.org/bugzilla/4.0/
modified Bugzilla/CGI.pm
Committed revision 7741.

Note You need to log in before you can comment on or make changes to this bug.