536873 - TB3.0 slow, takes a long time to load HTML messages with View Message Body as Original HTML, sits at "Loading Message ..." or presents blank screen

Status: RESOLVED DUPLICATE of bug 545126

(Thunderbird :: Message Reader UI, defect)

Description

[Mike]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0

I click on an 80k message that contains an attached image (that's the 80k), and it takes 12 seconds to load. I click on a 300k message that consists of an inline image and it takes 14 seconds to load. Testing and timing several messages tells me that any html message that contains images, whether attached or inline, loads very slowly. The image size appears to have only a small effect.

Reproducible: Always

Steps to Reproduce:
1. Click on an email in the list pane that contains an image.
2. Have a sip of coffee or coke while waiting for it to load in the message pane.

Actual Results:  
The email took an average of 10 to 15 seconds to load.

Expected Results:  
The email should load in a second or 2 max. In TB2 it did.

Comment 1

[Mike]

Btw, it doesn't matter what folder the message is in. I even tried moving a 180k message to a new folder all by itself and it still took 12 seconds to load when I clicked on it, so it's not corruption causing the problem.

Comment 2

[Ludovic Hirlimann [:Usul]]

Can you save one of those messages in .eml format and attach it to this bug using the add attachment link on this page ?

Is your issue still present when you start Thunderbird in -safe-mode (http://kb.mozillazine.org/Safe_mode) ?

Comment 3

[Mike]

I've already changed back to V2 for now. Is it possible to install V3 as a totally separate program so I can do testing with it while still using V2 for actual use?

I will attach one of those emails that I'll save from my version 2.0.0.23 Tbird. If it matters which version it's saved from, let me know and I'll reattach it after I reinstall V3.

Comment 4

[Mike]

Attachment: One of my slow loading emails, but saved from V2 not 3 since I switched back.

Comment 5

[Ludovic Hirlimann [:Usul]]

Ok the attachment load pretty fast on my machine in V3.

(In reply to comment #3)
> I've already changed back to V2 for now. Is it possible to install V3 as a
> totally separate program so I can do testing with it while still using V2 for
> actual use?

yes just install TB3 in another place with the install wizard. You can't launch both at the same time. I'm just curious as where it loads faster in safe-mode.

Comment 6

[Mike]

I reinstalled V3 in a separate location, made a copy of my main profile directory, and set up a profile using that directory so I could test without affecting my actual emails.

Brought it up normal first and it loaded slow. Restarted in safe mode and it still loaded slow.

I did another test. I created a new blank profile, set up 1 account on it, made a "storage" folder under local folders, then copied one of my slow loading emails to there. When I launched in that profile and clicked on that folder, then that message, it loaded in less than a second.

From that, I can conclude that the new version runs into trouble caused by my massive collection of email accounts, folders, and files. 

It still must be a bug because I've upgrade through a few V2's and the problem didn't pop up until V3.

I did delete all msf files in my copied profile folder before starting, so they got recreated. I'm thinking maybe I should delete some of the files in the root of my profile.
Here's all the files in the profile root:
XUL.mfl
xpti.dat
XPC.mfl
virtualFolders.dat
urlclassifier3.sqlite
signons.sqlite
session.json
secmod.db
prefs.js
pluginreg.dat
permissions.sqlite
panacea.dat
mimeTypes.rdf
mailViews.dat
localstore.rdf
key3.db
history.mab
global-messages-db.sqlite
folderTree.json
extensions.rdf
extensions.ini
extensions.cache
cookies.sqlite
compreg.dat
compatibility.ini
cert8.db
abook.mab

Which of these should I NOT delete? I'll delete all others and see if that fixes it.

Comment 7

[Mike]

Oops, that's the list from my blank profile. Here's the one from my full copy.
25 panacea.dat
25 XUL.mfl
abook.mab
blocklist.xml
cert8.db
compatibility.ini
compreg.dat
cookies.sqlite
cookies.txt
downloads.rdf
downloads.sqlite
extensions.cache
extensions.ini
extensions.rdf
folderTree.json
global-messages-db.sqlite
history.mab
impab.mab
impab-1.mab
impab-2.mab
impab-3.mab
impab-4.mab
impab-5.mab
impab-6.mab
impab-8.mab
junklog.html
key3.db
localstore.rdf
localstore-safe.rdf
Mail.rar
mailViews.dat
mimeTypes.rdf
panacea.dat
permissions.sqlite
persdict.dat
pluginreg.dat
prefs.js
prefs.js2
secmod.db
session.json
signons.sqlite
signons.txt
signons3.txt
training.dat
urlclassifier2.sqlite
urlclassifier3.sqlite
virtualFolders.dat
XPC.mfl
xpti.dat
XUL.mfl

Comment 8

[Mike]

I did another test. 

Since the problem could be due to the volume of mail stored, I deleted all files under Local Folders so the program would recreate the necessary inbox, trash, and outbox. I started the program and this time when I clicked on the large email, it loaded in 5 seconds. Emptying the local folders sped it up, but it's still a bit slow, so next I emptied my main business storage folder (on my copy of course) and started TB3 and this time the same message loaded in just under 2 seconds.

That almost looks like the more mail the program has stored, the slower it loads html emails, which could make sense, but then that would mean V2 is faster when dealing with more mail.

Comment 9

[Wayne Mery (:wsmwk)]

I'm not quite prepared to declare this confirmed or a canonical bug for http://gsfn.us/t/o85n but there is definitely a problem somewhere. But no speed problem when display body changed from Original Html to Simple Html, or with a new profile and Original Html.  I don't know why it's better with a new profile, but Borsuk confirms this.

This is with phishing disabled, testcase in this bug and one Borsuk gave from gsfn. Both load slowly with high cpu - 5-6 sec and 8-9 sec respectively. Perhaps slowness is even worse on slower machines or machines using USB.  And Borsuk reports the issue does not exist in v2. 

Issue is widely reported in forums.

related bug reports may be (I haven't investigated them)
 Bug 545126 -  HTML Mail View very slow (45 sec - 1min)
 Bug 542747 -  TDB Eating up the disk when try to open a mail with many images

Comment 10

[Wayne Mery (:wsmwk)]

actually reports include cases worse than a few seconds. Some reports measure in minutes.

Comment 11

[Wayne Mery (:wsmwk)]

Attachment: santized borsuk testcase from http://gsfn.us/t/o85n

Comment 12

[Wayne Mery (:wsmwk)]

happens in safe mode, so I am comfortable confirming.

The primary symptoms so far us high cpu and status bar has "Loading Message ...".
But after having been in safe mode most of the time I no longer see Loading Message.  I just see high cpu and a blank screen until the entire message is rendered - in other words, zero chrome.

Comment 13

[Wayne Mery (:wsmwk)]

sorry, drugged up. last post I hope.  To summarize:
- see this with 3.0 and 3.1 in old profile in safe mode
- do not see this in v2
- doesn't happen in new profile using any branch
- doesn't seem to be phishing related (I set 3 phishing prefs off)
- workaround: display body as Simple HMTL

Leaving for others to test:
- I wasn't able to test with Thunderbird 3.2 because that latest nightly is crashing
- does it happen only with profiles originally created in v2?  
- if not, what is the cause? AB related? (don't think so because in one of my 3.1 tests in an new profile, I loaded up my two biggest ABs)

IMO, this is bad enough UX to want for 3.1. Perhaps not block, but certainly wanted.  Ported back to 3.0 if possible

Comment 14

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

I just tried it and mac Thunderbird 3.0.2 candidate build 1 doesn't bork on https://bugzilla.mozilla.org/attachment.cgi?id=427638
i.e. the HTML email opens with no delay (images are broken though)

Comment 15

[Mike]

I just tested the switching email view option. Tried simple html and plain text, and it didn't help in my case. ("View" "Message Body As" options)

I'm still using Tbird 2 and probably will for a while by the looks of it.

Comment 16

[Wayne Mery (:wsmwk)]

Mike, odd. maybe we have more than one issue here. 

Is your account imap or pop?  Do you run antivirus software?
If you have time, please have a run through https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems

Comment 17

[Mike]

I use POP and Comodo antivirus.

I think what we need are a few people having slow loading emails try the message body viewing options to see what percentage it makes a difference for.

I've thought about trying from scratch, set up a new profile, create the email accounts 1 by 1, then just copy over all the emails without the msf files, but that would take a few hours which I don't really have yet. Plus, we have another little bug regarding the folder columns settings and until that gets changed, no way I'm going to take time to manually changed all my folder column settings.

I see TB3 as having 2 big (to me) bugs, so I'm sticking with TB2 for now.

I did some memory testing and first, the program was using 500MB the whole time, so I disabled all addons and restarted, then the program used 45MB. I clicked on a slow loading message and CPU went to 45% but memory didn't change. My computer is Intel Pentium D 3.2GHz with 3GB RAM running XP SP3.

I moved that email to a folder of its own and still used CPU and time to load it, so it can't have anything to do with having to read the whole mail file to get the email contents. I disabled Comodo and still slow load.

Summary:
Viewing as plain text with no active addons and no antivirus in a folder all by itself - still slow loading.

Comment 18

[Joe Sabash [:JoeS1]]

One of the problems with using an eml file as a testcase, is that they never
really 100% show the original composition.

FWIW the eml opens for me in < 5 seconds.

I really don't think that what I see accurately renders the original composition
View source shows many instances of html background image in table.

Example:
background="file:///C:/DOCUME~1/JOESAB~1/LOCALS~1/Temp/borsuk-gsfn.eml?fetchCompleteMessage=true&amp;part=1.1.4&amp;filename=bg_lblue_left.gif"><br>
          </td>
          <td><br>
          </td>
          <td colspan="3" bgcolor="#e0e6f0" height="1"><br>
          </td>
          <td><br>
          </td>
          <td
background="file:///C:/DOCUME~1/JOESAB~1/LOCALS~1/Temp/borsuk-gsfn.eml?fetchCompleteMessage=true&amp;part=1.1.5&amp;filename=bg_lblue_right.gif"><br>
          </td>
        </tr>
        <tr>
          <td
Those should tile out to the dimension of the table.
Where are they..I don't see them.

The CID protocol is not supported in the editor, and poorly supported in layout
for mailnews. This could be a contributing factor here.

One thing might be tried for those affected by this bug:
Maybe XUL.mfl file is at fault here TB2 Vs. TB3

Someone who is affected, might try deleting it in it in the TB3 profile.
(when Tb is not active of course)

Comment 19

[Wayne Mery (:wsmwk)]

Mike, thanks for doing that. Only through testing can we be sure those items aren't involved.  The fact that you had 500MB before safe mode and 45MB after indicates there is at least a problem with one of your add-ons.  But not surprising that it doesn't change the message loading time or cpu.

Joe, FWIW I have the original message forwarded to me and I created the sanitized testcase from it. All I can say is performance of what he sent me matches what I see with the eml.

Comment 20

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to comment #19)
> Mike, thanks for doing that. Only through testing can we be sure those items
> aren't involved.  The fact that you had 500MB before safe mode and 45MB after
> indicates there is at least a problem with one of your add-ons.  But not
> surprising that it doesn't change the message loading time or cpu.
> 
> Joe, FWIW I have the original message forwarded to me and I created the
> sanitized testcase from it. All I can say is performance of what he sent me
> matches what I see with the eml.

(In reply to comment #19)

Tried opening the sanitized EML in Windows 7 Thunderbird 3.0.1 and again it opened very quickly again with the same broken images as the Mac. The sanitized EML appears to be a very garden variety HTML email subscribe notice. 
> Mike, thanks for doing that. Only through testing can we be sure those items
> aren't involved.  The fact that you had 500MB before safe mode and 45MB after
> indicates there is at least a problem with one of your add-ons.  But not
> surprising that it doesn't change the message loading time or cpu.
> 
> Joe, FWIW I have the original message forwarded to me and I created the
> sanitized testcase from it. All I can say is performance of what he sent me
> matches what I see with the eml.

Comment 21

[Wayne Mery (:wsmwk)]

seamonkey 2.0.3 by my counting of seconds seems worse. 
and doesn't improve with view original or view plain.
it did, however, improve with a combo of not display attach inline, and view plain

not display inline also helps thunderbird

as previously noted, this happens with phishing disabled. however, might be good to do more QA in this area, and also have a look at  Bug 441481 -  excessive IO on urlclassifier3.sqlite

see also http://forums.mozillazine.org/viewtopic.php?f=39&t=1644095&p=8326505&hilit=slow+display+html#p8326505 and http://forums.mozillazine.org/viewtopic.php?f=39&t=1645445&p=8221935&hilit=slow+display+html#p8221935 (which claims to be resolved by disabling email scams)

Comment 22

[Dan Mosedale (:dmosedale, :dmose)]

We're resetting the blocking flag for 3.1 on this bug and instead setting the wanted-thunderbird+ flag. We have too many blocking-3.1 bugs, to the point where it doesn't mean much, and managing the list is making it hard to actually work on closing bugs, which helps no one.

Thunderbird 3.1's primary purpose is to allow us to offer a prompted major update to Thunderbird 2 users, to ensure their continued ability to safely use Thunderbird.  Thunderbird 2 is built on an outdated version of Gecko, and our long-term ability to maintain the users' safety for Thunderbird 2 users is limited.

If you think this bug meets the requirements below, please renominate with a detailed explanation of how it meets the following two criteria, and we will reconsider.  To qualify, this bug must either:

a) make the upgrade experience from TB2 very painful for a large number of users

or

b) be a new, reproducible, severe quality issue (eg dataloss, frequent crashes)

Just because this bug doesn't block TB3.1 doesn't mean it can't or won't make the release.  Once they're done with their blockers (if any), we encourage developers to keep working on non-blocking bugs, and to try to land them as early in the cycle as possible, as non-blocking bugs will become increasingly difficult to land in the later stages of the cycle.

Comment 23

[Wayne Mery (:wsmwk)]

Mike, please test and report whether your issue matches bug 545126. For that bug bug...
2009-06-19-03-comm-1.9.1 works fine
2009-06-20-03-comm-1.9.1 problem 

see bug 545126 comment 29. you can get the 1.9.1 builds from here ftp://ftp.mozilla.org/pub/thunderbird/nightly/2009/06/

xref http://forums.mozillazine.org/viewtopic.php?f=31&t=1707075&p=8993355#p8993355

Comment 24

[Mike]

Same for me.

2009-06-19-03-comm-1.9.1 worked fine
2009-06-20-03-comm-1.9.1 loaded very slowly

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #24)
> Same for me.
> 2009-06-19-03-comm-1.9.1 worked fine
> 2009-06-20-03-comm-1.9.1 loaded very slowly

Mike, do use MS Win's Offline Files(CSC, client-side caching) for Tb's profile directory or Tb's mail directory?
> http://technet.microsoft.com/en-us/library/bb457104.aspx#EEAA

Comment 26

[Mike]

(In reply to comment #25)
> Mike, do use MS Win's Offline Files(CSC, client-side caching) for Tb's profile
> directory or Tb's mail directory?
> > http://technet.microsoft.com/en-us/library/bb457104.aspx#EEAA

No

Comment 27

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #15)
> Tried simple html and plain text, and it didn't help in my case. ("View" "Message Body As" options)
(In reply to comment #26)
> No

Hmm... It sounds different issue from bug 545126 even though same regression range as bug 545126...
('sits at "Loading Message ..." or presents blank screen' in bug summary sounds particurarity of this bug...)

To Mike(bug opener):

Does your problem occur on any HTML mail with image?
Or your problem occurs with some specific HTML mails only? (with mail like you attached to this bug only?)

Where is your Tb's profile directory and/or mail directory placed?
Local HDD? Or file server? Or USB?
If file server, what kind of NFS do you use? SMB which is standard of MS Win? (usually "NetBIOS over TCP", if MS Win's standard file sharing).
If USB, what is version of USB interface of your PC? What is version of USB which the USB memory device supports?

(In reply to comment #12)
> happens in safe mode, so I am comfortable confirming.

Wayne Mery, does it mean you could reproduce original problem of this bug?
If yes, how? In what environment? With what test case? With what STR? With what setup?

(In reply to comment #21)
> seamonkey 2.0.3 by my counting of seconds seems worse. 
> and doesn't improve with view original or view plain.
> it did, however, improve with a combo of not display attach inline, and view plain
>
> not display inline also helps thunderbird

Wayne Mery, is it same phenomenon/problem as original problem of bug summary of this bug and comment #0 of this bug?

Comment 28

[Mike]

(In reply to comment #27)
> To Mike(bug opener):
> 
> Does your problem occur on any HTML mail with image?
> Or your problem occurs with some specific HTML mails only? (with mail like you
> attached to this bug only?)
> 

Anything with inline images, not with images as attachments.

> Where is your Tb's profile directory and/or mail directory placed?
> Local HDD? Or file server? Or USB?
> If file server, what kind of NFS do you use? SMB which is standard of MS Win?
> (usually "NetBIOS over TCP", if MS Win's standard file sharing).
> If USB, what is version of USB interface of your PC? What is version of USB
> which the USB memory device supports?
> 

Local hard drive, but not default location. I chose the location on drive D. On my system, C is Windows only and all other programs are on D.

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #28)
> Anything with inline images, not with images as attachments.
> Local hard drive, (snip)

Can you check with test case attached to bug 545126 comment #82?
> Attachment 435098 [details]
> Case-02 : multpart/related, 40 <img src="cid:....">, 89589 bytes(21*4096+3573)
> (save as .eml, drag the .eml file thread pane of a folder of Tb 3)
This test case took 10 seconds to display in my environmnet.

Comment 30

[WADA:World Anti-bad-Duping Agency]

FYI.

>(Your bug summary)
> TB3.0 slow, takes a long time to load HTML messages
> with View Message Body as Original HTML,
> (a) sits at "Loading Message ..." or (b) presents blank screen

Above phenomena can easily be observed by merely creating many mail folder files, with TB 3.0.x, with HTML mail of medium mail size/medium number of <img>, with local profile directory/local mail folder file, without Offline Files(CSC).
(1) Create local mail folder named "AAAA", create a subfolder under "AAAA", 
    create local mail folder named "ZZZZ" under same account,
    show "Order Received" column(offset of mail in local mail folder file),
    mail-1 : copy a small text mail to "ZZZZ" (offset=0) 
    mail-2 : copy above HTML mail of Case-2 to "ZZZZ" (offset=xxx),
    (save as .eml, drag the .eml file to thread pane of "ZZZZ") 
    terminate Tb
(2) Create file of 0000, 0001, 0002, ..., 9999 in directory of ...\AAAA.sbd
    (by BAT, by Script, by hand, ...)
(3) Restart Tb(takes long due to rebuild-index of 10,000 mail folders).
    If restart takes too long, reduce file number in ...\AAAA.sbd, please.
(4) Click folder of "ZZZZ"
(5) When no mail is displayed, click mail-2(mail of Case-02)
    => (b) presents blank screen
    Take a short coffee break until all images in HTML mail is displayed.
(6) Click mail-1 => small mail-1 is displayed.
    click mail-2(mail of Case-02)
    => (a) sits at "Loading Message ..."
    Take a short coffee break until all images in HTML mail is displayed.
(7) Terminate Tb, delete directory of ...\AAAA.sbd, restart Tb.

See bug 545126 and read thru the bug for problem and cause(s), please.
  this bug(bug 536873) :
     Local profile directory/mail folder file. Offline Files(CSC) is not used.
     Slowness in display of HTML mail with img is not exposed by small mail.
  bug 545126 :
     Remote profile directory/mail folder file. Offline Files(CSC) is used.
     Slowness in display of HTML mail with img is exposed by very small mail.

Comment 31

[David :Bienvenu]

WADA, any objections to just marking this as a dup of bug 545126?

Comment 32

[WADA:World Anti-bad-Duping Agency]

No objection.