importxml.pl stops after attachment

RESOLVED WORKSFORME

Status

()

RESOLVED WORKSFORME
9 years ago
8 years ago

People

(Reporter: ruud.siebierski, Unassigned)

Tracking

Details

Attachments

(3 attachments)

(Reporter)

Description

9 years ago
User-Agent:       Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.5.30729)
Build Identifier: Bugzilla 3.4.2 on perl 5.8.8

Did an export of 2 bugs in BZ 3.0 by using xml.cgi
1 bug was with an attachment (the first one in the xml file), second without attachment. Tried to import bugs using importxml.pl in BZ 3.4.2. Seemed that after the process_attachment handler was called and executed successfully, everything just stopped, process_bug was never called. Removed the bug with attachment, then it worked fine. So my guess was process_attachment was wrong (this was first time i saw Perl code...). Tried to localize the problem.
After changing the code like shown below, it worked (pls dont ask why, something to do with references maybe?)

        my $date = $attach->field('date');
        my $desc = $attach->field('desc');
        my $type = $attach->field('type');
        my $attachid = $attach->field('attachid');
        my $filename = $attach->field('filename');
        my $attacher = $attach->field('attacher');

    $attachment{'date'} = format_time( $date, "%Y-%m-%d %R" ) || $timestamp;
    $attachment{'desc'}       = $desc;
    $attachment{'ctype'}      = $type || "unknown/unknown";
    $attachment{'attachid'}   = $attachid;
    $attachment{'ispatch'}    = $attach->{'att'}->{'ispatch'} || 0;
    $attachment{'isobsolete'} = $attach->{'att'}->{'isobsolete'} || 0;
    $attachment{'isprivate'}  = $attach->{'att'}->{'isprivate'} || 0;
    $attachment{'filename'}   = $filename || "file";
    $attachment{'attacher'}   = $attacher;


Reproducible: Always

Steps to Reproduce:
1. create bug.xml with 2 bugs, 1 with attachment
2 [review]. perl -T importxml.pl -v -v bug.xml
3. check dbase
Actual Results:  
no bug is imported

Expected Results:  
2 bugs are imported

Original snippet importxml.pl source code of process_attachment

....
    format_time( $attach->field('date'), "%Y-%m-%d %R" ) || $timestamp;
    $attachment{'desc'}       = $attach->field('desc');
    $attachment{'ctype'}      = $attach->field('type') || "unknown/unknown";
    $attachment{'attachid'}
    $attachment{'attachid'} = $attach->field('attachid');
    $attachment{'ispatch'}    = $attach->{'att'}->{'ispatch'} || 0;
    $attachment{'isobsolete'} = $attach->{'att'}->{'isobsolete'} || 0;
    $attachment{'isprivate'}  = $attach->{'att'}->{'isprivate'} || 0;
    $attachment{'filename'}   = $attach->field('filename') || "file";
    $attachment{'attacher'}   = $attach->field('attacher');
....

Updated

9 years ago
Keywords: qawanted

Comment 1

9 years ago
What version of XML::Twig are you using?
(Reporter)

Comment 2

9 years ago
Did a ./checksetup.pl with the following output (see below for complete output).
I would like to add that, the import is now not working in the process_bug method. 

Checking for            XML-Twig (any)     ok: found v3.32


complete output:

Checking perl modules...
Checking for              CGI.pm (v3.21)   ok: found v3.48
Checking for          Digest-SHA (any)     ok: found v5.47
Checking for            TimeDate (v2.21)   ok: found v2.23
Checking for            DateTime (v0.28)   ok: found v0.50
Checking for   DateTime-TimeZone (v0.71)   ok: found v1.01
Checking for                 DBI (v1.41)   ok: found v1.52
Checking for    Template-Toolkit (v2.22)   ok: found v2.22
Checking for          Email-Send (v2.00)   ok: found v2.198
Checking for          Email-MIME (v1.861)  ok: found v1.863
Checking for Email-MIME-Encodings (v1.313)  ok: found v1.313
Checking for Email-MIME-Modifier (v1.442)  ok: found v1.444
Checking for                 URI (any)     ok: found v1.35

Checking available perl DBD modules...
Checking for              DBD-Pg (v1.45)    not found
Checking for           DBD-mysql (v4.00)   ok: found v4.013
Checking for          DBD-Oracle (v1.19)    not found

The following Perl modules are optional:
Checking for                  GD (v1.20)   ok: found v2.44
Checking for               Chart (v1.0)    ok: found v2.4.1
Checking for         Template-GD (any)     ok: found v1.56
Checking for          GDTextUtil (any)     ok: found v0.86
Checking for             GDGraph (any)     ok: found v1.44
Checking for            XML-Twig (any)     ok: found v3.32
Checking for          MIME-tools (v5.406)  ok: found v5.427
Checking for         libwww-perl (any)     ok: found v2.033
Checking for         PatchReader (v0.9.4)   not found
Checking for          PerlMagick (any)      not found
Checking for           perl-ldap (any)      not found
Checking for         Authen-SASL (any)      not found
Checking for          RadiusPerl (any)      not found
Checking for           SOAP-Lite (v0.710.06)  not found
Checking for         HTML-Parser (v3.40)   ok: found v3.55
Checking for       HTML-Scrubber (any)      not found
Checking for Email-MIME-Attachment-Stripper (any)      not found
Checking for         Email-Reply (any)      not found
Checking for         TheSchwartz (any)      not found
Checking for      Daemon-Generic (any)      not found
Checking for            mod_perl (v1.999022) ok: found v2.000004

Comment 3

9 years ago
This is the second complaint I have seen like this. I suspect it has something to do with recent changes to XML::Twig. Try downgrading to 3.29 and see if the issues still remain.
(Reporter)

Comment 4

9 years ago
Downgraded to 3.29, same problem.
With which version did you test the import script?

[root@localhost bugzilla]# ./checksetup.pl
* This is Bugzilla 3.4.2 on perl 5.8.8
* Running on Linux 2.6.18-92.el5 #1 SMP Tue Jun 10 18:49:47 EDT 2008

Checking perl modules...
Checking for              CGI.pm (v3.21)   ok: found v3.48
Checking for          Digest-SHA (any)     ok: found v5.47
Checking for            TimeDate (v2.21)   ok: found v2.23
Checking for            DateTime (v0.28)   ok: found v0.50
Checking for   DateTime-TimeZone (v0.71)   ok: found v1.01
Checking for                 DBI (v1.41)   ok: found v1.52
Checking for    Template-Toolkit (v2.22)   ok: found v2.22
Checking for          Email-Send (v2.00)   ok: found v2.198
Checking for          Email-MIME (v1.861)  ok: found v1.863
Checking for Email-MIME-Encodings (v1.313)  ok: found v1.313
Checking for Email-MIME-Modifier (v1.442)  ok: found v1.444
Checking for                 URI (any)     ok: found v1.35

Checking available perl DBD modules...
Checking for              DBD-Pg (v1.45)    not found
Checking for           DBD-mysql (v4.00)   ok: found v4.013
Checking for          DBD-Oracle (v1.19)    not found

The following Perl modules are optional:
Checking for                  GD (v1.20)   ok: found v2.44
Checking for               Chart (v1.0)    ok: found v2.4.1
Checking for         Template-GD (any)     ok: found v1.56
Checking for          GDTextUtil (any)     ok: found v0.86
Checking for             GDGraph (any)     ok: found v1.44
Checking for            XML-Twig (any)     ok: found v3.29
Checking for          MIME-tools (v5.406)  ok: found v5.427
Checking for         libwww-perl (any)     ok: found v2.033
Checking for         PatchReader (v0.9.4)   not found
Checking for          PerlMagick (any)      not found
Checking for           perl-ldap (any)      not found
Checking for         Authen-SASL (any)      not found
Checking for          RadiusPerl (any)      not found
Checking for           SOAP-Lite (v0.710.06)  not found
Checking for         HTML-Parser (v3.40)   ok: found v3.55
Checking for       HTML-Scrubber (any)      not found
Checking for Email-MIME-Attachment-Stripper (any)      not found
Checking for         Email-Reply (any)      not found
Checking for         TheSchwartz (any)      not found
Checking for      Daemon-Generic (any)      not found
Checking for            mod_perl (v1.999022) ok: found v2.000004
We have the same Problem with Bugzilla 3.4.2

Environments:
SuSE Linux Enterprise Server 10 SP2 x86
XML::Twig 3.32 (also tested 3.29 and 3.33)
XML::Parser 2.34
expat 2.0.0

SuSE Linux Enterprise Server 9 SP4 x86
XML::Twig 3.32
XML::Parser 2.34
expat 1.95.7


The output of importxml.pl (with the -v -v switch) is:
> Reading xml
> Parsing tree
> Parsing attachments

The sub function process_attachment() gets executed till the end but nothing seems to happen after there (the skript exits with return code 0).

It makes no difference if the attachment data is directly embedded into the xml (encoded) or in an external file (with --attach_path defined). The binary- & metadata read by the sub function looks sane for me.

The fix from the bug description doesn't work here.

Comment 6

9 years ago
Please attach an XML file which causes trouble. I will try locally using your files. Make sure there is no sensitive data in the file you upload. ;)
OS: Linux → All
Hardware: x86 → All
Version: unspecified → 3.4.2
Created attachment 411650 [details]
bug xml with (external) attachment
Created attachment 411651 [details]
attachment of bug xml
(In reply to comment #6)
> Please attach an XML file which causes trouble. I will try locally using your
> files. Make sure there is no sensitive data in the file you upload. ;)

The XML file is attached. The import of the same file w/o the <attachment> section works flawless.
I cannot reproduce the issue with the XML file you attached. The bug and its attachment are correctly imported.

The only problem I found with your XML file is its first line:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

There must be a whitespace before ?>.

I'm using XML::Twig 3.32 and XML::Parser 2.36.
Keywords: qawanted

Comment 11

9 years ago
Exact same problem here.  When attempting bug moves, bugs with attachments don't get imported, bugs without attachments get imported fine.  Bugzilla 3.4.4, XML::Twig 3.32.

Comment 12

9 years ago
I tested an install of Bugzilla 3.2.5 on a freshly-installed system, with all Perl modules brought up to Bugzilla 3.4-required versions.  Importing of a bug with attachments was working fine.  Then, I upgraded to Bugzilla 3.4.4 (via CVS), and without any further changes to any Perl modules, importing bugs w/ attachments immediately stopped working.  Seems to be a Bugzilla bug.

Comment 13

9 years ago
After a lot more digging, it appears this was somehow caused by the changes to Bugzilla/Util.pm's format_time() in 3.4.  When I replaced the call to format_time() in importxml.pl with just filling in a static dummy value, attachments were imported fine.  I did make sure I had the latest version of DateTime, DateTime::Locale, and DateTime::TimeZone installed, but that didn't change anything.

I ended up extracting the format_time() function from Bugzilla 3.2, inserting it as a new function into 3.4's importxml.pl (and use'ing Date::Parse and Date::Format which it requires).  Bug moving now seems to work fine as before.

Comment 14

9 years ago
I should have also mentioned that, of course, I modified process_attachments() in importxml.pl to call my new format_time() (the one lifted from 3.2) instead of the one from Bugzilla/Util.pm.  I left the rest of the format_time() calls in importxml.pl alone, they don't seem to cause trouble.
Bob, could you attach an XML file which is imported correctly with the old format_time() function but not with the new one?
Keywords: qawanted

Comment 16

9 years ago
Created attachment 446610 [details]
Bug w/ attachment that fails import on stock 3.4.4

Here's one that always failed for us before I made the date-related changes I described.  All testing was done with Bugzilla 3.4.4 and we still use that version.
I still cannot reproduce your problem with the XML file you attached. I tested on both Bugzilla 3.4.6 and 3.7.1.

Comment 18

9 years ago
Problem happens for me on both Red Hat Enterprise Linux 4 (i386) and 5 (x86_64).  Haven't tested with Bugzilla 3.4.6, though.

Comment 19

8 years ago
Support list questions about importxml.pl exiting after only one import are being redirected to this bug, which may or may not be related.  I, for one, cannot import multiple bugs from a single XML file, although a coworked who supplied the XML files can on his laptop.
I definitely cannot reproduce the issue. Putting this bug out of the QA radar.
Keywords: qawanted

Updated

8 years ago
Duplicate of this bug: 607622

Comment 22

8 years ago
I confirm, it's the same thing on bugzilla 3.4.6
I fixed this by mading the changes described on the comment 14 and 15. 
i can now use the move function of bugzilla successfully. But this is not the case when i run importxml.pl from the command line with any xml file from STDIN. 

Fida
Bugzilla 3.4 is now restricted to security fixes only. I definitely cannot reproduce your issue. Reopen if you can reproduce the problem with Bugzilla 4.0 or newer.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.