568810 - slow reading incredibly large emails

Status: RESOLVED DUPLICATE of bug 599119

(Thunderbird :: Message Reader UI, defect)

Description

[Lothsahn]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4) Gecko/20100526 Thunderbird/3.1

I often receive extremely large emails due to build failures from our automated build server.  These emails are incredibly large (approximately 10MB EACH, without attachments).

Even though I use "make available offline" so that the messages are already downloaded from the server, clicking on the email hangs Thunderbird for about 5 seconds (on a high-end modern machine) while the email is rendering.

This behavior was significantly faster in Thunderbird 3.  It appears that something in Thunderbird 3.1 is significantly slower in this area.

Also slow is multi-selecting a number of emails to delete them--probably due to attempting to render the emails as well.



Reproducible: Always

Steps to Reproduce:
1. Create a 10MB email and send it to yourself
2. Click on the email to view the email
3. Wait 5 seconds, approximately, on a high end machine.  The entire Thunderbird user interface is hung during this time.
4. Email renders
Actual Results:  
Email renders in about 5 seconds, and hangs the user interface while rendering.  it is not possible to select another email in the folder or do anything while this is occurring.

Expected Results:  
The email should render quickly, as they did in Thunderbird 3.

We use an IMAP server on exchange

My machine is a Dell M6400 with a quad-core, 2.53 GHZ Core2 DUO processor and 8GB of ram.

Comment 1

[Joe Sabash [:JoeS1]]

What kind of content plaintext,HTML or both.
If HTML what is the nature of the HTML, many tables, embedded images etc.

TB3.1 is based on a different Gecko core, so it's quite possibly a core rendering problem.

A few Perf problems were fixed in 3.1 so if you could identify more specific issues, that would be great.
Or if the data in the subject mail is not "sensitive" you could send me one.
I'm using POP3 so that would at least eliminate IMAP/syncing as a cause.

Comment 2

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Actual Results:  
> Email renders in about 5 seconds, and hangs the user interface while rendering.

Does it occur with same mail in local mail folder(non IMAP, "Local Mail Folder")?
If HTML, does it occur with View/Message Body As/Plain Text and View/Display Attachments Inline=OFF, with same mail in local mail folder?

Comment 3

[Lothsahn]

Attachment: Inbox for Local Folders (invalid HTML)

This is the inbox.  However, when I tried to sanitize the message, I corrupted the HTML.  I will write a short program to leave the HTML intact but remove all message data.

Comment 4

[Lothsahn]

Joe:

I have provided a sanitized email for you.  Unfortunately, I can't provide the original.  This email also takes a long time to load (more like 10 secnods).  However, because the HTML is invalid, I can't guarantee it's taking the same code path now.

I'm going to work on sanitizing the email and providing you a new testcase where the HTML is intact but the message data is gone.  However, I am seeing a performance issue with this example.

Rather than sending you a 10MB email, I thought it would be better to send you my local folders inbox file.  The only email contained in that file is the affected email.

Let m e know if you need anything else.

Comment 5

[Lothsahn]

WADA:

1) The problem DOES occur in the local folders
2) The problem DOES occur when viewing as plain text
3) The problem does occur with display attachments inline=off

You can look at the attachment (replacing your existing inbox file (and deleting the inbox's index) to reproduce the problem.

Comment 6

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #1)
> TB3.1 is based on a different Gecko core, so it's quite possibly a core rendering problem.

Bug 419937?
> Bug 419937 Loading large text page extremely slow

Comment 7

[Lothsahn]

WADA:

Read through the other bug, and I don't have enough details to say whether or not this is a duplicate of that one.  It very possibly could be, or they might have different root causes.

It looked to me like the other bug was triggered by one or more lines that was very very wide.  I'm not aware of any very very long lines in the email (it just has a LOT of lines).  That being said, there might be one line somewhere in the email that is extremely long.

Comment 8

[Wayne Mery (:wsmwk)]

Lothsahn, 
To what extent does speed improve when using v3.1.5 or higher from ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.2/ or setting plugin.disable", true and restarting thunderbird?
(Bug 599119 may offer improvement)

Comment 9

[Lothsahn]

I use thunderbird for my work email, so I'd prefer not to install an alpha/beta release.

I tried to find plugin.disable in the config editor, but was unable to find it.  Is that the exact name?

Finally, I would think that this is not the issue, because it's directly related to the email size.  For instance, most emails open for me within a second (could be snappier, but I view it as acceptable).  I would assume that rescanning the plugin directory would be a constant cost, not related to the email size.


On the really large emails, the two I tested took:
Thunderbird 3.1.4:
Message 1 (8:21AM):
36 seconds
37 seconds

Message 2 (10:07AM): (this email is smaller than Message 1)
6 seconds
7 seconds

If thunderbird 3.1.5 comes out within a couple weeks, I'll post the numbers for the same emails at that time.  However, these large emails do get purged after a month (otherwise my inbox would become HUGE), so there's a small window.

I'd be happy to benchmark once 3.1.5 is released...

Comment 10

[Wayne Mery (:wsmwk)]

I understand the skepticism. The key here is that you think this is a 3.1 regression. As far as I know, and can tell, there are no other bugs a reported as 3.1 regressions for message display speed [1].  So it's not unreasonable that at least *part* of what you are seeing may be bug 599119. Note: modest sized messages stored locally (for example your message 2) should display almost instantaneous. 

Conversely, if what you are seeing is not an 3.1 regression, it is almost certainly a duplicate of one of the bugs in [1].

ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-1.9.2/ builds are neither alpha nor beta. They are release 3.1.4 plus only high quality, controlled integration patches, which becomes v3.1.5. hence you see they are named 3.1.5pre.  (And they have few thousand users which helps)

plugin.disable must be created (forgot to mention) - right click in the editor pane and pick new > boolean.


[1] reported message display speed bugs. if all the bugs are marked correctly, none are 3.1 regressions in the list. Including Bug 563677 for example, which is a 3.0 regression.
https://bugzilla.mozilla.org/buglist.cgi?value1-1-0=addreess&type1-0-0=nowordssubstr&short_desc=large%20html&field0-0-0=short_desc&type0-0-1=substring&field0-0-1=keywords&type1-0-1=allwordssubstr&resolution=---&type1-1-0=nowordssubstr&query_format=advanced&field1-1-0=keywords&value1-0-0=delete%20address&short_desc_type=anywordssubstr&value0-0-1=perf&type0-0-0=anywordssubstr&value0-0-0=slow&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=short_desc

Comment 11

[Lothsahn]

Wayne:

I retested with plugin.disable and got the following results:

Message 1 (huge):
19 seconds

Message 2 (very large):
3 seconds

So clearly this helps significantly and I am clearly wrong about it not being a fixed cost per message. :)

I also notice that general UI responsiveness is greatly improved, especially for normal sized messages.  While message loading times used to be subsecond for me, I now find that they are near instantaneous--I would estimate 250ms or less to load an email that is normal-sized.

19s isn't great, but...  for reference, Outlook can't even open these messages as it crashes. :)  I guess this bug can be marked as resolved, although it would be *nice* if the emails opened faster than 19s...

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to comment #11)
> I also notice that general UI responsiveness is greatly improved, especially
> for normal sized messages.  While message loading times used to be subsecond
> for me, I now find that they are near instantaneous--I would estimate 250ms or
> less to load an email that is normal-sized.
> 
> 19s isn't great, but...  for reference, Outlook can't even open these messages
> as it crashes. :)  I guess this bug can be marked as resolved, although it
> would be *nice* if the emails opened faster than 19s...

thanks for testing that. yeah, reduction of 17 seconds is a great improvement. it's still rather odd that 3.1 was worse for you than 3.0.

As for the remaining slowness, there are bugs which cover several types of large messages.  And even though you are presently mostly "satisfied", I suggest you report a new bug with a testcase message you still see slowness after the next major release (3.2, 3.3 or something along those lines).