1505245 - TB hangs in mozTXTToHTMLConv when viewing e-mail with XML-like plaintext attachment with a very long line (300KB)

Status: NEW

(Thunderbird :: General, defect)

Description

[Matt]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0

Steps to reproduce:

Go to "Sent" folder.
Open a sent email that has an attachment.


Actual results:

Completely hangs. Only happens for some emails and not others. No idea why.


Expected results:

Should open the email.

I tried in the old version (50.9.2) and the new (60.x) and it does the same in both.

Comment 1

[Wayne Mery (:wsmwk)]

How much CPU is being used when it hangs?
Does the problem go away if you do Help > Restart with Add-ons disabled ?

Comment 2

[Matt]

(In reply to Wayne Mery (:wsmwk) from comment #1)
> How much CPU is being used when it hangs?
> Does the problem go away if you do Help > Restart with Add-ons disabled ?

OK, I just discovered that it doesn't completely hang. I timed it with a stop watch. It hangs for 4 min 58 seconds. After that, the email finally opens and the hang ends. The CPU goes to 100% during this time. That was for a file with a 1.3MB attachment. The duration of the hang seems to vary depending on the size of the attachment, but not linearly. Perhaps O(n^2) with attachment size.

Restarting with Add-ons disabled does not help. (I don't have any add-ons in any case).

I also just noticed, when I scroll down the messages, the attachments are showing in-line at the bottom of the messages.
But they are not text files. Thunderbird should not be dumping the content into the message. Perhaps that is the problem. The file type is .sdlxliff

Comment 3

[Matt]

OK, I did a bit more digging. Although Thunderbird does not attempt to display .doc or .docx or .ppt. or .pttx or .zip files inline, for some reason it tries to dump .sdlxliff files inline. Why is it doing that? The .sdlxliff file structure is an xml wrapper around a base64 encoded binary payload, so displaying the content as text is meaningless. 

I can uncheck "Display Attachments Inline" in the view menu, but then I lose all the normal attachments that should be viewed (i.e image files).

Comment 4

[Matt]

More digging, and now I understand it is not exactly a display problem (although having the entire app freeze for 5 minutes is pretty bad user experience). The problem is that Thunderbird is setting the attachment to the wrong mime type.
I looked at the message source, and for the attachment it has:
Content-Type: text/plain; charset=UTF-8;

But it should be:
Content-Type: application/octet-stream;

This is an email in my sent folder, so the mime type was set by Thunderbird.
So the next question is how is Thunderbird getting the wrong mime type?
I checked mimeTypes.rdf, and there was nothing in there. It is Windows OS,
so is it picking up the wrong mime type some other way?

I assume this mime type issue is also causing problems for the people who are receiving my emails.
(This whole thing started because recipients weren't getting the attachments, and I went into my 
sent folder to check that I had actually attached the file). So, how do I fix this? It is no longer
just a display issue, but a core mail functionality issue.

Comment 5

[Jorg K (CEST = GMT+2)]

Can you supply such an attachment or bad message please.

Comment 6

[Matt]

The data in the file is under NDA and confidential to customer, so I cannot send the actual file.
I have tried creating dummy data, and it is surprisingly difficult to do. I'll list what I have tried, since this might help with resolving:

1. I took a zip file and renamed it to .sdlxliff file and sent. Thunderbird sets content type to "application/octet-stream", and there are no problems with the file.
2. I took a largish C source file I had, renamed it to .sdlxliff and sent. Thunderbird sets content type to "text/plain; charset=UTF-8;", the same as the problem .sdlxliff. However, it does not cause Thunderbird to hang when displayed inline.
3. I found a largish xml file, renamed it to .xdlxliff and send. Thunderbird sets content type to "text/xml". Again, Thunderbird does not hang when displaying this file.

I should probably note at this point, the files in 2 and 3 are supposed to be human readable and contain linefeed/carriage returns, whereas the problem file is in xml format but is only meant to be machine readable, and does not contain any excess whitespace. I'm not sure if this is why Thunderbird both fails to detect it as xml and also hangs when opening it.

I'll try creating some dummy file, but it may take a while.
Also, in the meantime, I found a solution. In the Windows registry, I went to HKEY_CLASSES_ROOT\.sdlxliff and added "Content Type"/"application/octet-stream". It seems that Thunderbird checks the registry for this.

OK, I managed to create a file that causes the problem. I made it a bit small so that it will only hang Thunderbird for 10 to 20 seconds. Now, how to attach it here?

Comment 7

[Matt]

Attachment: ThunderbirdFail2.docx.sdlxliff - The file to attach to cause the problem

Comment 8

[Matt]

Attachment: Thunderbird Fail Attachment.eml - Email Thunderbird creates and which make Thunderbird hang

I edited the eml file to remove my email address from it. Everything else is as-is.

Comment 9

[Jorg K (CEST = GMT+2)]

Thanks, I can confirm everything you said.

Note that Thunderbird uses Mozilla platform code, so that does the sniffing into what type of document it may be.

IMHO, Content-Type: text/plain; isn't wrong since the document *is* text and human-readable, so I think the more interesting question is why it hangs so long on display. We can display 500KB images faster than that.

Since this will be an intricate investigation (for a single user who already found a workaround), please don't expect fast action here.

Note: This needs to be triaged into the right component.

Comment 10

[Wayne Mery (:wsmwk)]

About a minute for me to load ~6000 lines. If 5 minutes for someone else then for all intents and purposes may as well be a hang.

Likely a duplicate, but it's not obvious to me what the distinguishing feature is of this testcase.

Comment 11

[Alfred Peters [:infofrommozilla]]

(In reply to Jorg K (GMT+1) from comment #9)
> human-readable, so I think the more interesting question is why it hangs so
> long on display. We can display 500KB images faster than that.

May be: Bug 564051

Comment 12

[Alfred Peters [:infofrommozilla]]

OK, this is a bit special.
70% of the time is spent in mozTXTToHTMLConv::ItMatchesDelimited(),
called by mozTXTToHTMLConv::NumberOfMatches():

<https://dxr.mozilla.org/comm-central/source/netwerk/streamconv/converters/mozTXTToHTMLConv.cpp#673>

NumberOfMatches() counts the number of closing tags in a line, when
an opening tag was found (Text tags like *bold*, _underline_ or /italic/ -
not HTML tags).

In a line of one to two hundred characters,
this is peanuts. But the last line of attachment 9023852 [details] has 294k characters
with >8k '/' in it.

Workaround: set the Pref "mail.display_struct" to false.
This gives a great performance boost.

What could we do:
We could only apply this function to lines <200 characters.
Or we don't apply it to attachments at all.

Comment 13

[Jorg K (CEST = GMT+2)]

I think the latter. Great investigation BTW.