366672 - importxml.pl doesn't work in commandline mode

Status: RESOLVED INVALID

(Bugzilla :: Bug Import/Export & Moving, defect)

Description

[Gunnar Wagenknecht]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Build Identifier: 

According to importxml.pl documentation it's possible to invoke it from the command line. However, this doesn't work because the XML is lost because of some Email/MIME related parsing which should not happen in command line mode

See importxml.pl around line 1181 and following:

# If the email was encoded (Mailer::MessageToMTA() does it when using UTF-8),
# we have to decode it first, else the XML parsing will fail.
my $parser = MIME::Parser->new;
$parser->output_to_core(1);
$parser->tmp_to_core(1);
my $entity = $parser->parse_data($xml);
my $bodyhandle = $entity->bodyhandle;
$xml = $bodyhandle->as_string;


Reproducible: Always

Comment 1

[Gunnar Wagenknecht]

This happens using Bugzilla HEAD (what will become Bugzilla 3.0 soon)

Comment 2

[Frédéric Buclin]

I cannot reproduce the bug. Could you attach an email which causes trouble?

Comment 3

[Gunnar Wagenknecht]

No email, just a simple XML file.

Comment 4

[Greg Hendricks]

I can't reproduce this either. I am doing the following:

$bugizilladir> ./importxml.pl -v -v ~/import.xml

import.xml is a file that I saved from exporting as xml from the show_bug page.

Is your bug data sensitive? Can you attach your xml file for us to test with?

Comment 5

[Greg Hendricks]

Ok, I think I see what is going on here. 

The Mime parsing is removing the first line of the xml up the the first <bug> tag. I am not sure what changed to make this so since this area has not changed since 2.22.

The attempt I made before was complainging that a bad param was sent to Bugzilla::User. This turned out to be because the exporter was not included in the xml since it had been stripped. 

Comment 6

[Greg Hendricks]

It seems that it will remove everything before the first blank line. As a work around, try removing all the blank lines from your xml (that are not nested in tags such as longdesc) and add a couple blank lines to the very top of the document.

Doing this allowed me to import my bugs successfully. I will try to nail down a fix for this.

Comment 7

[Frédéric Buclin]

Weird, I have no such problem. Saving a bug in the XML format and then calling importxml.pl on it creates the bug successfully. No error thrown.

Comment 8

[Gunnar Wagenknecht]

Could it be some CPAN module that is causing the problem here?

The problem is that on my system the MIME parsing screwed up the XML. I tested it with putting print statements before and after. The XML was mangled in a way that made it impossible to parse.

A simple xml was even truncated completely, i.e. $xml was empty after the MIME processing. 

Comment 9

[Gunnar Wagenknecht]

Attachment: Example XML

Here is some sample XML. This didn't work out of the box. But when I commented the MIME parsing part it was imported successfully.

Comment 10

[Frédéric Buclin]

No such problem with version 5.420 of MIME::Parser:

perl -MMIME::Parser -we 'print $MIME::Parser::VERSION;'
5.420

Note that my XML files start with
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "http://localhost/bugzilla222/bugzilla.dtd">

and not with <bugzilla.

Comment 11

[Greg Hendricks]

The problem here is that your XML file is invalid. A valid xml file starts with <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

MIME::Parser therefore thinks it is part of the headers and removes everything before the first line. If you start your file with the <?xml line it will work.

Comment 12

[Lieven de Cock]

Attachment: xml export failing to import

Comment 13

[Lieven de Cock]

I can confirm this bug, I ran into it on 2.22.1.
It is the mime parsing stuff that breaks the xml. In my case it removed :
- the <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
- the doc type
- <bugzilla
So basically it makes the xml invalid, and because the <bugzilla ...> tag is no longer there we don't end up anymore at the init(), and more problems arise (see below)

This was my original, more detailed, analysis : posting it here :
I have done my tests on bugzilla 2.22.1.
I think I found the reason of those uninitialized values (see below  for
the original posting)
Line 558/599 : my $url = $urlbase . "show_bug.cgi?id=";  ===>
complaining about    $urlbase
Line 594 :     $comments .= "\n\n--- Bug imported by $exporter_login "; 
===> complaining about    $exporter_login
Why those complaints : well they never got a value.
They ought to get a value in sub init(), but we never made it in there,
so it could not get the attributes from the root.
So the question is why   did we not get into the init(). As we see near
the end of the file the following line code :
my $twig = XML::Twig->new(
    twig_handlers => {
        bug        => \&process_bug,
        attachment => \&process_attachment
    },
    start_tag_handlers => { bugzilla => \&init }
);
$twig->parse($xml);

meaning if we see the tag "bugzilla" call init for it, and similar when
we see the tags bug and attachment. The xml export file has as root tag
<bugzilla>, so ORIGINALLY it was present.

Let me first copy another little piece of code :
# Read STDIN in slurp mode. VERY dangerous, but we live on the wild side  ;-) 
local ($/);
$xml = <>;

# If the email was encoded (BugMail::MessageToMTA() does it when using
UTF-8),
# we have to decode it first, else the XML parsing will fail.
my $parser = MIME::Parser->new;
$parser->output_to_core(1);
$parser->tmp_to_core(1);
my $entity = $parser->parse_data($xml);
my $bodyhandle = $entity->bodyhandle;
$xml = $bodyhandle->as_string;

when one print out the $xml after the line "$xml = <>", then we see the
correct content of our original file staring with the xml declaration,
doctype and <bugzilla> tag, but after those MIME::Parser stuff and the
assignment :  $xml = $bodyhandle->as_string , the $xml is not so correct
anymore the xml declaration and doctype AND <bugzilla> tag is gone.
Especially the fact that the <bugzilla ....> tag is gone results in no
call to the init() method.
When I remove those MIME::Parser lines (especiallly "$xml =
$bodyhandle->as_string;", according to the comments it's for stuff
coming in through email, and in my case the xml file was handed over at
the command line ==> so this should probably be conditional == don't do
it when no through email ???) then the import works (well I think the
import could use some enhancements, but that's another story  ;-)   ).

So, since you are the experienced bugzilla man, I hope my analysis is of
any help to you and this can be correctly fixed.

I have attached the test export I used.

kind regards,
Lieven

Comment 14

[Greg Hendricks]

This has actually been fixed. See bug 368071