548277 - [mozTXTToHTMLConv/libmime] Should run plaintext recognizer only over text/plain. XML attachment(s) (or other text/* that are not text/plain) cause TB to hang

Status: NEW

(MailNews Core :: MIME, defect)

Description

[jason.turner]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

Selecting an email to view the content causes TB to crash if the email contains large XML attachments

Reproducible: Always

Steps to Reproduce:
1. receive an email with large XML attachment(s)
2. select the email in the TB viewer
3.
Actual Results:  
Beachball, maxed out CPU for a short while, then back to normal but beachball stays. Have to kill the process

Expected Results:  
Allowed the email content to be viewed and the attachments to be downloaded

Comment 1

[Ludovic Hirlimann [:Usul]]

Jason can you share an example email with us ?

Comment 2

[jason.turner]

Unfortunately not, the email and especially the XML are confidential to one of my clients. And it will take me a while to get the XML files off their systems by alternative means anyway. All I can say about the attachments is that they're probably quite big.

Apart from the XML attachments, there were three Word doc attachments (I'm sure these don't have anything to do with the problem) and the rest is plain text.

I tried sending myself an email with a single small XML file attached and this caused no problems, but since the TB viewer shows the attachment contents after the text of the message, perhaps there's something in XML attachments which confuses the viewer when it opens them (I noticed a couple of posts in various blogs about XML attachments crashing TB - try googling: thunderbird crash xml attachments).

I'll send myself some mails with different XML attachments when I have some time, probably not until Friday or the weekend. I'll update here if I have fresh info ...

Comment 3

[jason.turner]

Unfortunately not, the email and especially the XML are confidential to one of my clients. And it will take me a while to get the XML files off their systems by alternative means anyway. All I can say about the attachments is that they're probably quite big.

Apart from the XML attachments, there were three Word doc attachments (I'm sure these don't have anything to do with the problem) and the rest is plain text.

I tried sending myself an email with a single small XML file attached and this caused no problems, but since the TB viewer shows the attachment contents after the text of the message, perhaps there's something in XML attachments which confuses the viewer when it opens them (I noticed a couple of posts in various blogs about XML attachments crashing TB - try googling: thunderbird crash xml attachments).

I'll send myself some mails with different XML attachments when I have some time, probably not until Friday or the weekend. I'll update here if I have fresh info ...

Comment 4

[jason.turner]

Just got hold of the attachment XML, tried sending myself an email with only one of these attached. Again crashed TB. I was right, the files are quite large - over 1.5Mb, UTF-8 encoding, no newlines at all (! perhaps it's causing some kind of overflow).

Hope this is a small help, I will try playing with different kinds of attachment when I have some time ...

Comment 5

[timeless]

can you try with windows?

we don't seem to have shark enabled builds of thunderbird/seamonkey, but we do have a symbol server for windows....

https://developer.mozilla.org/En/How_to_get_a_stacktrace_with_windbg
use http://symbols.mozilla.org/thunderbird/ instead of /firefox/ ....

Comment 6

[jason.turner]

Attachment: XML test files used to demonstrate bug

See last post ...

Comment 7

[jason.turner]

Hmmm... haven't touched windoze in years, and now that I've got all my friends to throw it out and use Linux or Mac OSX instead I'm not likely to be able to try it out on somebody else's windoze box either. Sorry.

Had a bit of time to try this out with different XML files, and the idea that the lack of newlines was causing the problem seems correct - two similar XML files, index1.xml and index2.xml were sent as attachments (I am sending these as tb_debug_xml.zip). Both are around 1.9Mb in length, index1.xml has no newlines, index2.xml is more conventionally laid out. TB previews the email containing the attached index2.xml correctly below the message text, but when opening the email containing the attached index1.xml, TB hangs every time.

Comment 8

[Mark Banner (:standard8)]

Attachment: Sample of Hang

I took two samples of the hang at slightly different times using a 3.0.4pre nightly (20100302) so the debug symbols should be good.

Here's the first.

Comment 9

[Mark Banner (:standard8)]

Attachment: Second sample of hang

Comment 10

[Mark Banner (:standard8)]

Oh and I forgot to say that just creating an email with this attachment, saving it as draft and then trying to view it causes the hang as well.

Comment 11

[Ben Bucksch (:BenB)]

The stack shows the plaintext link recognizer. As far as I know, this would only happen, if the XML file was *not* attached, but pasted in the body. Attached files do not, to my knowledge, run through the recognizer - not even plaintext files. Yet, the description specifically says "attachments".

I don't see a real bug here, FWIW. It's a "large" file, and it takes a long time to process. It may be very long, but email is simply not made for large files. Ever tried to attach a video to an email?

Comment 12

[Ben Bucksch (:BenB)]

Looking at the files, index1.xml is 2 MB large and doesn't seem to contain any linebreaks, it's a single 2 MB line. That's uncommon, and unwise, and breaks many software. I've seen several editors hang on such things, too. Our libmime does line-based processing. The link recognizer therefore is fed one (whole) line at a time. It's using nsString and does a lot of operations on the strings, under the assumption that it's usually about 80 chars, or a maximum of 1000 chars (RFC821), or a paragraph. It's not surprising that feeding it a 2 MB line breaks both the recognizer and various nsString functions.

However, what puzzles me most is that we run the recognizer over the attachment. Maybe we should just blacklist text/xml.

Comment 13

[Ben Bucksch (:BenB)]

Another idea would be to chop the lines into parts of 1000 or 10000 chars, if they are longer than that.

Comment 14

[jason.turner]

XML is machine-readable markup - linebreaks serve no purpose and for efficiency some XML generators leave them out, especially when the files are large. I work with this kind of XML very often.

What would be really cool is if the recogniser rendered [compliant] XML in the same way Firefox does with collapsible tags. That would mean that once it had recognised the attachment as XML it would be processing it by tags and not by lines.

In any case, it needs to be able to deal with attachments containing streams of characters like this. I run TB on a brand new MacBook Pro 3.06GHz which is just about as poky as they come. And if it hangs processing attachments like this it's not just "taking a long time to process", it's broken code.

Comment 15

[Andrew Sutherland [:asuth] (he/him)]

Blacklisting sounds appropriate; no point spending a ton of time optimizing a case that doesn't make a lot of sense, likely affects only a handful of users, and even those users probably aren't looking for link love.

Comment 16

[Mark Banner (:standard8)]

(In reply to comment #11)
> The stack shows the plaintext link recognizer. As far as I know, this would
> only happen, if the XML file was *not* attached, but pasted in the body.
> Attached files do not, to my knowledge, run through the recognizer - not even
> plaintext files. Yet, the description specifically says "attachments".

Just to confirm: the XML file *was definitely* attached and not inserted.

Comment 17

[Ben Bucksch (:BenB)]

Yes, I saw it too.
Actually, instead of blacklisting, we should only run the recognizer on text/plain attachments and not on all of text/*. For example, it should not parse text/x-diff either.

Comment 18

[Felix]

I can confirm this bug in TB3.1 on Windows XP SP3.

In my case the XML file is 1.09MB, and it contains no line breaks.

Backstory: 

This is an EDI file which is normally delivered automatically to our customers systems.
However, in this case there was a query, so they emailed it back to me.
They probably use Outlook, which let them send it, whereas when I receive it in TB the program hangs.

When I look at the email they have sent me in GoogleMail, I can see that the file is attached rather than being 'in line'.

Hope you can fix this!

Comment 20

[derrynj]

I've just run into this bug... 10 years later. I have 1-2MB Word XML files that are presenting with the same issue. Any chance this will be looked at again?