Closed Bug 545126 Opened 15 years ago Closed 14 years ago

HTML Mail View very slow (45++ sec) for pop and local folders

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Version:
1.9.1 Branch
Platform:
x86
Windows XP
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(blocking-thunderbird3.1 rc1+, thunderbird3.1 beta2-fixed, blocking-thunderbird3.0 -, thunderbird3.0 wontfix)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
blocking-thunderbird3.1 --- rc1+
thunderbird3.1 --- beta2-fixed
blocking-thunderbird3.0 --- -
thunderbird3.0 --- wontfix

People

(Reporter: g.lamken, Assigned: Bienvenu)

References

(Blocks 2 open bugs,
URL
)

Details

(Keywords: perf, regression, Whiteboard: [gs][ameliorating fix landed for 3.1 b2])

Attachments

(8 files)

example email with Problem in HTML-View on old Pentium 4 machines
6.74 KB, text/plain
Details
NSPR log(all:5) for view HTML mail attached to this bug
60.23 KB, text/plain
Details
Process Monitor log(file access by Tb 3.0.3) for view HTML mail attached to this bug, with local profile/local mail folder file
1.55 MB, text/plain
Details
Case-00 : multpart/mixed, 4 attached image/jpeg, 9284 bytes(2*4096+1092)
9.07 KB, text/plain
Details
Case-01 : multpart/related, 4 <img src="cid:....">, 9390 bytes(2*4096+1198)
9.17 KB, text/plain
Details
Case-02 : multpart/related, 40 <img src="cid:....">, 89589 bytes(21*4096+3573)
87.49 KB, text/plain
Details
wip patch
10.86 KB, patch
Details | Diff | Splinter Review
[checked in] proposed fix for 3.1
8.33 KB, patch
standard8
: review+
standard8
: superreview+
standard8
: approval-thunderbird3.1+
Details | Diff | Splinter Review 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

Some HTML Mail freezes Thunderbird for a while.
Mostly in large email folders but not necessarily a large email.
Emails came from Lotus-Notes-Client and contains only few text and some spacer GIF for layout.

Effect also occurs in '-safe-mode' of Thunderbird.
No anti-virus programm is activ.
Yes, Profile is in network folder but this is probably not the reason.
Text-Viw of same email is OK.
Only HTML view appears as problem.

CPU-use graph in Windows-Task-Manager shows very high peaks for a while.


Reproducible: Always

Steps to Reproduce:
1. create a larger Emailfolder (about 200 plus xx)
2. ask to get HTML-mails from different Lotus Notes User
3. Open Task-Manager for CPU-use
5. switch preview of emails an notice what happens
Actual Results:  
Email is not shown for a while. 
Can't change to another mail or function.

Expected Results:  
show the email content in preview window.
Forgot to mention.
Reproduced this behaviour on two other Desktop with same configuration.
I have seen this complaint before, but have never been able to duplicate it.
Could you provide a testcase (EML) properly sanitized for privacy.
Either as an attachment here, or to my email address ?
Keywords: perf
Component: General → Message Reader UI
QA Contact: general → message-reader
Version: unspecified → 3.0
You got in confidence some anonymized examples.
Unfortunately this Problem still exist with Thunderbird 3.0.3.
Even with two different brand new Core Duo notebooks it is no fun to open HTML-email, which we got from Lotus-Notes users.
Under this circumstances TB 3 is not usable for professional use.
For the record, the examples provided did not produce lag time anywhere near
45 seconds to render. Further the lotus-notes compositions were very sloppy
with space key being used to position elements, and other redundancies.

Certainly, we cannot control the quality of mail that appears in our inbox,
but without a testcase that can be attached here, this bug will get little
traction.

I'll keep looking for a suitable testcase from mail and newsgroups for proper
public posting and analysis.
Local mail folder(POP3, Local Folders)? IMAP folders?
If IMAP, "offline use=on"? Or off?

Can you check next?
(1) Save the mail to .eml file
(2) Open Outlook Express, Drag&Drop the .eml to a mail folder, view the mail
(3) By Tb 3, create a dummy POP3 account, create a local ditectory,
    select the created directory at Server Settings/Local Directory:,
    and restart Tb 3(mandatory).
(4) By Tb 3, open the dummy POP3 account, open Inbox,
    Drag&Drop the .eml to Inbox or thread pane or Inbox,
    view the mail(original HTML)
(5) Create mail folder named "Inbox2" at the dummy POP3 account.
    Copy the mail in Inbox to Inbox2.
    Click Inbox. (force close of file named Inbox2)
(6) Open file named Inbox2(not Inbox2.msf) by text editor.
    Change all <img to <ximg (kill <img> tag), an save. 
(7) At Folder Properties of Inbox2, "Rebuild Index".
(8) View mail in Inbox2 (Original HTML, View/Display Attachments Inline=Off).
(9) How many attachments are displayed in attachment pane?

What is time to take display at (2), (4), (8)?
tested with my last example I sent you on Fr March, 12th 
with Manual Stopwatch, about
(2) 000:00:00.575
(4)a 000:02:12.645 original mail
(4)b 000:02:28.538 anonymized mail you got
(8) 000:00:00.555
(9) attachments (spacer GIFs) in attachment pane are only shown if text view is used. But than in both cases inbox and inbox2. I was quite astonished that inbox2 shows attached Gif in the attachement pane. The email in inbox2 was fast shown in HTML-view. But the IBM-Logo was vanished, therefore the <img to <ximg manipulation did work.
> I was quite astonished that
> inbox2 shows attached Gif in the attachement pane. The email in inbox2 was fast
> shown in HTML-view. But the IBM-Logo was vanished, therefore the <img to <ximg
> manipulation did work.

With a short look into the email source I understand that attached GIFs themself are not effected by manipulations of an img tag.
But the img tag manipulation actually hat a positive effect for rendering time.
(In reply to comment #6)
> Outlook Express(probably .pst is placed on local HDD) 
>  (2) 000:00:00.575
> Tb, held on local HDD, original, with <img>
>  (4)a 000:02:12.645 original mail
>  (4)b 000:02:28.538 anonymized mail you got
> Tb, held on local HDD, without <img>
>  (8) 000:00:00.555

O.E. doesn't display <img> as default? Or you used text mode display instead of HTML mode disply? Or external image is used and O.E. doesn't display external image by default?

How many <img> tags are used in the HTML mail with IBM-Logo?
Inline image? (<img src="cid:...">
Or external image? (<img src="http:...">,<img src="file:...">
(In reply to comment #8)
> (In reply to comment #6)
> > Outlook Express(probably .pst is placed on local HDD) 
> >  (2) 000:00:00.575
> > Tb, held on local HDD, original, with <img>
> >  (4)a 000:02:12.645 original mail
> >  (4)b 000:02:28.538 anonymized mail you got
> > Tb, held on local HDD, without <img>
> >  (8) 000:00:00.555
This whole scenario is held on local HDD.
> O.E. doesn't display <img> as default? Or you used text mode display instead of
> HTML mode disply? Or external image is used and O.E. doesn't display external
> image by default?
O.E. (Outlook Express?) displays <img> in HTML-view, and IBM-Logo is visible.
But O.E. is used in Offline mode, because it contains only a dummy account.
> 
> How many <img> tags are used in the HTML mail with IBM-Logo?
> Inline image? (<img src="cid:...">
> Or external image? (<img src="http:...">,<img src="file:...">

Here is part of the source
Content-Type: multipart/related; boundary="=_related 0035E069C1257618_="
X-UIDL: Sh?!!p'Z"!'R/!!Z)M!!

This is a multipart message in MIME format.
--=_related 0035E069C1257618_=
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable

[...]

<tr>
<td colspan=3D4><img src=3Dcid:=5F1=5F06A926C406A923080035E066C1257618>
<td valign=3Dtop>
<td valign=3Dtop>
<tr>

[...]

--=_related 0035E069C1257618_=
Content-Type: image/gif
Content-ID: <_1_06A926C406A923080035E066C1257618>
Content-Transfer-Encoding: base64

R0lGODlhmwIJAIAAAP///wdIiCH5BAAAAAAALAAAAACbAgkAAAJzhI+py+0Po5y02ouz3rz7D4bi
SJbmiabqyrbuC8fyTNf2jef6zvf+DwwKh8Si8YhMKpdMVuAJjUqn1Kr1is1qt9yu9wsOi8fksvmM
TqvX7Lb7DY8HmvS6/Y7P6/f8vv8PGCg4SFhoeIiYqLjI2Oj4CPlSAAA7

[...]
> How many <img> tags are used in the HTML mail with IBM-Logo?

Seven,
all of them are related like above.
>  (4)b 000:02:28.538 anonymized mail you got

I've received the mail data via mail from you.

I can't reproduce problem. Mail was displayed at a glance, although I could see CPU 10%-20%(Core2 Duo, so 20%-40% if single CPU) just a while(less than 1 sec) upon mail display with "Original HTML".
Following is message pane display when CTRL+A is pressed at message pane.
(Part1.x is attachment name displayed when "View/Message Body/Plain Text")

> --------------------- <== Part1.5 667*9 pixel gif, black line
>
>              IBM-Logo <== Part1.6   
>
> --------------------- <== Part1.7 667*9 pixel gif, black line
> XXX  ...             +-+ 
> Sitz ...             | |  ---...--- <= Part1.8 1000*1 pixel gif, white line
>                      +-+ 
>                       A
>                       |
>                       +-- reversed box for &nbsp;

Possibly due to the very wide table raw of wide spanned cell(667pixel wide) and 1000px white line in a cell.

Table raw of "XXX..., Sitz... + IBM-Logo" is perhaps "signature part". Probably mail sender generated garbage of non-visible &nbsp; and white gif line of 1000px*1px in the signature data.
g.lamken@web.de, edit mail data and remove table cell of 1000px white line gif. Still very slow and high CPU utilization?
(as cellspan is used at many places, layout is changed by the remove)
> I can't reproduce problem. Mail was displayed at a glance, although I could see
> CPU 10%-20%(Core2 Duo, so 20%-40% if single CPU) just a while(less than 1 sec)
> upon mail display with "Original HTML".

Sorry following information was only given in on of my direct mails.
>> My machine is an old P4 3000Mhz, 1GB Ram.
>> My Core Duo Laptop is a bit faster of course.

Many of our Desktops in our Company are Intel P4 ones and have the same HTML-view problems, what is hindering them to migrate to TB3.

> Following is message pane display when CTRL+A is pressed at message pane.
Can't follow this.
CTRL+A means in most cases I know (including TB) "select all"

How did you get that view above?

> remove table cell of 1000px white line gif.
> Still very slow and high CPU utilization?
Yes.
I also tried this step by step.
Only the first spacer.
Only the logo. etc.

Only if I destroy the image tag (<ximg) or if I delete the attached images, the HTML-View is very fast.
Even the single IBM-Logo is a performance problem.

I've created a HTML-email with TB using the GIF from the problematical example.
Even with the well formed HTML from TB this performance problem occurs.
I'll send this example as attachment in Bugzilla

--------
by the way:
if instead of an email folder an account-item is selected, and I switch from plain text to html view I get an error in the error console:
Error: gFolderDisplay.view.dbView is null
Source: chrome://messenger/content/msgMail3PaneWindow.js
Line: 1047

new Bug Item?
--------
needed example to test on slow machine.
on current systems this problem don't occurs
FYI
most of HTML-Spam (or Advertisment) mail are uncritical displayed in HTML-view.
But unfortunately this example of problem stands for many mails from our official contacts (very big companies). I don't think that they will change their email layout, because we insist to use TB.
(In reply to comment #13)
> > Following is message pane display when CTRL+A is pressed at message pane.
> Can't follow this.
> CTRL+A means in most cases I know (including TB) "select all"

You are right. CTRL+A is equivallent to Edit/Select/All.

> How did you get that view above?

Display the sent mail with View/Message Body As/Original HTML, and CTRL+A == Edit/Select/All.
White line of 1000px width is last <img>(Part 1.8) in the HTML.

> Even the single IBM-Logo is a performance problem.

I see.
The new testcase renders in about 3 seconds for me.
I'm using a P3-s 1.4 gig so I don't think raw CPU speed is your problem.
It might not be a bad idea to see if the problem is related to your graphics card.
There have been perf problems on specific graphics cards in the past.
You might also check if the mfg has a newer driver available.
g.lamken@web.de, can you check next?
(1) Tools/Options/Advanced/Network & Disk Space, Disk Space, "Clear Now"
    Improved?
(2) At Command Prompt; (delete files under Temp directory)
       DEL %temp%\*.* /S /F /Q
    Improved?
(In reply to comment #18)
> g.lamken@web.de, can you check next?
> (1) Tools/Options/Advanced/Network & Disk Space, Disk Space, "Clear Now"
>     Improved?
No

> (2) At Command Prompt; (delete files under Temp directory)
>        DEL %temp%\*.* /S /F /Q
>     Improved?
Sorry, but this command would delete too much.
Some of my paid work processes need TEMP too.

I reopened the test set on my old notebook (Pentium M 1.7 GHz).

First Result:
Test-Emails are displayed acceptable fast.
That means that on my desktop my dummy-account works under a different condition than it was supposed to do. I think it is not worth to check this, because we have now a better test environment on that notebook.

But (!)
that notebook works with a local copy of my mail folders.
I changed the path in the properties to my profile on the network drive, and got that described problem in html view again.
Therefore something is wrong with my dummy account.

Concerning our discussion.
There must be a change from TB2 to TB3 concerning the way TB is loading mail from network drives.
I remember a BUG discussion, where someone noticed something what fits to this.
But I'm not on the same technology knowledge level to repeat this in the right term, and I can't find the BUG No either.

This is what I remember:
TB3 talks now to the network in smaller bits than TB2 did. This probably leads to communication problems.

Maybe parts of the GIF didn't reach TB3 just in time, when they are needed to be rendered?
(In reply to comment #19)
> > (2) At Command Prompt; (delete files under Temp directory)
> >        DEL %temp%\*.* /S /F /Q
> >     Improved?
> Sorry, but this command would delete too much. Some of my paid work processes need TEMP too.

i see. Check files(not directory) named nsXXX.tmp, nsXXX-N.tmp, nsXXX.eml, nsXXX-N.eml etc., please. Are garbage file which seems to be created by Tb seen in %temp%? See Bug 389132 for performane problem in the past due to many nsmail-N.tmp in %temp. If Tb uses %temp% for decoding of base64 encoded data and/or decompress of image data, similar problem may occur.

> This is what I remember:
> TB3 talks now to the network in smaller bits than TB2 did.
> This probably leads to communication problems.

Probably Bug 539389. It's main reason why I requested you to check with local mail folder file, although I thought the bug is irrelevant to your problem because the bug is for write operation, not for read operation.
> Bug 539389 Very slow file manipulation deleting or moving messages
> (profile stored on synchronized network folder).
> Tb3 requests write for each line of mail data. Should use buffering.
> Check files(not directory) named nsXXX.tmp, nsXXX-N.tmp, nsXXX.eml,
> nsXXX-N.eml etc., please. Are garbage file which seems to be created by Tb seen
> in %temp%? See Bug 389132 for performane problem in the past due to many
> nsmail-N.tmp in %temp. If Tb uses %temp% for decoding of base64 encoded data
> and/or decompress of image data, similar problem may occur.

Cleaned up %temp%.
On my old notebook %temp% was already clean.
Unfortunately this had no positive effect.

I've also started a radical experiment and uninstalled completely everything what was combined with Mozilla (and Netscape what we used once long time before).

Only mail folder from network (none of any other old Profile folder or files) are reused with the new installed TB.

The behaviour is still the same. HTML-View is slow.
(In reply to comment #21)
> The behaviour is still the same. HTML-View is slow.

HTML mail only related issue of Tb only?
Or HTML mail with image part(usually base64 encoded) related issue of Tb only?
No problem, if local HTML file like next, and if Firefox or other browsers?
> <html><head></head><body>
> <br><img src="abc/def/xyz.jpg">
> <br><img src="abc/def/xyz.gif">
> <br><img src="abc/def/xyz.png">
> </body></html>

No problem if mail like next, with View/Display Attachment Inline=On, with View/Message Body As/Original HTML?
> multipart/mixed
>   text/plain or text/html <= mail body
>   image/jpeg <= attachment part, but it can be displayed in inline by Tb
>                 - no name parameter in Content-Type: header
>                 - or, If Content-Disposition: header exists,
>                       - not Content-Disposition:attachment (inline, or wrong string)
>                       - and, no filename parameter in Content-Disposition: header
>   image/gif  <= attachment part (same as above)
>   image/png  <= attachment part (same as above)

Is "<img src=cid:...>" in "<table> in HTML mail source" relevant?
I think comment 22 might be on to it.

However, you all are spending lots of time on this when perhaps simply finding the regression range might pinpoint the change that caused the problem, and be quicker. (this is a regression, right?)
http://www.rumblingedge.com/2009/02/24/howto-find-regression-windows-through-manual-binary-search/
(In reply to comment #23)
> I think comment 22 might be on to it.
> 
> However, you all are spending lots of time on this when perhaps simply finding
> the regression range might pinpoint the change that caused the problem, and be
> quicker. (this is a regression, right?)
> http://www.rumblingedge.com/2009/02/24/howto-find-regression-windows-through-manual-binary-search/

Described Problem occurs after update from TB 2.0.0.23 to TB 3.0.1
It still existing under TB 3.0.3
> Described Problem occurs after update from TB 2.0.0.23 to TB 3.0.1
> It still existing under TB 3.0.3
Our method to have our mail-profile on a network driver works almost 10 years, started with Netscape. Problems with this kind of emails didn't appear before but of course layouts and HTML-Mail-style might have changed over the times ;-)
(In reply to comment #22)

> HTML mail only related issue of Tb only?
Yes or don't know
E.g. I cannot simulate a network environment for Outlook Exp.

> Or HTML mail with image part(usually base64 encoded) related issue of Tb only?
How is this be tested?
> No problem, if local HTML file like next, and if Firefox or other browsers?
No Problem. I stored problematical emails and images as HTML and changed the links to the gif.
Firefox has no problems with this.

> No problem if mail like next, with View/Display Attachment Inline=On, with
> View/Message Body As/Original HTML?
> > multipart/mixed
> >   text/plain or text/html <= mail body
> >   image/jpeg <= attachment part, but it can be displayed in inline by Tb
> >                 - no name parameter in Content-Type: header
> >                 - or, If Content-Disposition: header exists,
> >                       - not Content-Disposition:attachment (inline, or wrong string)
> >                       - and, no filename parameter in Content-Disposition: header
> >   image/gif  <= attachment part (same as above)
> >   image/png  <= attachment part (same as above)

Sorry this is too cryptic for me ;-) 

> 
> Is "<img src=cid:...>" in "<table> in HTML mail source" relevant?

How can I prove this?
(In reply to comment #24)
> (In reply to comment #23)
> > I think comment 22 might be on to it.
> > 
> > However, you all are spending lots of time on this when perhaps simply finding
> > the regression range might pinpoint the change that caused the problem, and be
> > quicker. (this is a regression, right?)
> > http://www.rumblingedge.com/2009/02/24/howto-find-regression-windows-through-manual-binary-search/
> 
> Described Problem occurs after update from TB 2.0.0.23 to TB 3.0.1
> It still existing under TB 3.0.3

That's good info, but we already know that. We need a regression range approaching 1 day time period. So you need to continue the procedure at 

"I went back further, trying trunk builds from Mozilla FTP using nightlies every year (backwards) …

Tried the build from 2007-12-01 (Shredder Alpha 1 was released in the early part of 2008)… No luck.

2006-12-01 does not have this bug. Now I’m getting somewhere…

2007-06-30 has this bug. Ah, a 6 month regression window, finally… but still a tad too large.

(The key point is to take intervals that divide themselves by half everytime – from 1 year, to 6 months, and now I’m going down to 3 months…) ..."
> trying trunk builds from Mozilla FTP using nightlies
> every year (backwards)

I assume that I can install those builds parallel without any side effects?
I need my desktop for daily work so otherwise it takes some time to find and configure a free test computer.
right. 
a) install each test build in a separate directory so that  you can retest if needed
b) so that you can keep v3 running, start your test thunderbird as  thunderbird.exe -P -no-remote  ... -P so you can choose a test profile, and -no-remote so that it doesn't complain that you have thunderbird v3 already running 

see http://kb.mozillazine.org/Run_multiple_copies_of_Thunderbird_at_the_same_time#Making_Thunderbird_behave_differently and http://kb.mozillazine.org/Command_line_arguments
(In addition to comment #29)

If you test on MS Win, download win32.zip build instead of installer build. You can test by Unzip only. If you test with different profile you are using daily and you use POP3 server, please don't forget to set "Leave messages on server" or not to define real POP3 account.

(In reply to comment #26)
> Sorry this is too cryptic for me ;-) 

Create an HTML mail(text/html only, no text/plain part), attach jpeg, png, gif files to the mail(no inline image by <img> in HTML mail).
See mail source of mail in Outbox of "Local Folders" after you create an HTML mail and execute "Send Later", please. 

> > Is "<img src=cid:...>" in "<table> in HTML mail source" relevant?
> How can I prove this?

Check with an HTML mail(text/html only, no text/plain part), with inline image by <img>(<img src="cid:...">), without any <table> in the HTML mail, please.
(In reply to comment #19)
> I reopened the test set on my old notebook (Pentium M 1.7 GHz).

> But (!)
> that notebook works with a local copy of my mail folders.
> I changed the path in the properties to my profile on the network drive,
> and got that described problem in html view again.
> Therefore something is wrong with my dummy account.

According to your Comment #6, "local mail folder or remote mail folder" is irrelevant to you problem.
What is difference between test of Comment #6 and test of Comment #19?

Many of "physical write per a line" occurs on pluginreg.dat and panacea.dat in profile directory becaus of known Bug 539389. But it should be same for any of HTML mail with image, HTML mail without image, and text/plain mail.
Difference between (a) "mail in local mail folder" and (b) "mail in remote mail folder" should be next only.
 (Read mail data, your test mail=6KB mail)
   (a) Read 4K + Read 2K from local file
   (b) Read 4K + Read 2K from remote file
 (Update of X-Mozilla-Status: if required)
   (a) Read 22 bytes and (re-)Write 22 bytes to local file
   (b) Read 22 bytes and (re-)Write 22 bytes to remote file
 (Update of .msf file)
   (a) Read/write of local .msf file
   (b) Read/write of remote .msf file
All other accesses to "profile" should be almost same for "mail in local folder" and for "mail in remote folder".

Note:
Write access by Tb3 to next files only was observed by viewing the test mail(log was obtained by Process Monitor on MS Win).
> "IRP_MJ_WRITE","C:\$LogFile","SUCCESS", ...
> "IRP_MJ_WRITE","C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\e866j8kl.yyy\Mail\a.a.a\Inbox","SUCCESS", ...
> "IRP_MJ_WRITE","C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\e866j8kl.yyy\Mail\a.a.a\Inbox.msf","SUCCESS", ...
> "IRP_MJ_WRITE","C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\e866j8kl.yyy\Mail\smart mailboxes\Inbox.msf","SUCCESS", ...
> "IRP_MJ_WRITE","C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\e866j8kl.yyy\panacea.dat","SUCCESS", ...
> "IRP_MJ_WRITE","C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\e866j8kl.yyy\pluginreg.dat","SUCCESS", ...

Did you restart Tb after "Server Settings/Local Directory:" setting change?
(if not, test result is misunderstood. see Bug 2654, please.)
Is program directory of Tb also placed on network drive?
(In reply to comment #31)
> According to your Comment #6, "local mail folder or remote mail folder" is
> irrelevant to you problem.
> What is difference between test of Comment #6 and test of Comment #19?
On our Notebook we usually use a copy of the remote Mail-folder.
The same email on local copy was displayed in acceptable time.
I changed in the properties of "Local Folder" the local Path to my network path.

> Did you restart Tb after "Server Settings/Local Directory:" setting change?
Yes.
Strange Task Manager entry.

Sorry but on the task manager I only watched the 'processes' tab or 'system'.
What I notice now on desktop and notebook, that alway, when TB3 needs time for HTML-view rendering, on application tab a second thunderbird icon appears. Both thunderbird application entries have status 'no response' for a while.
The second application entry vanishes and appears several times.
During this on 'processes' tab only one TB entry exists.
(In reply to comment #27)
Used "win32".zip often from 'unsigned' folder. I hope that is ok.

TB 2.0.0.24 (20100228) works fine
TB 3.0a1 works fine
TB 3.0a2us-en (shredder) works fine
TB 3.0a2de (shredder) works fine
TB 3.0b1.Build1.de works fine
TB 3.0b2.Build1.en-us works fine
TB 3.0b2.Build2.de works fine
TB 3.0b3.Build1.en-us slow html view on specific HTML-mails
TB 3.0b3.Build1.de slow html view on specific HTML-mails
TB 3.0b4.Build4.de slow html view on specific HTML-mails
TB 3.0rc1.Build1.de slow html view on specific HTML-mails
TB 3.0 (stable release) slow html view on specific HTML-mails
(In reply to comment #34)
> (In reply to comment #27)
> Used "win32".zip often from 'unsigned' folder. I hope that is ok.

yup, zips are fine


> TB 3.0b2.Build1.en-us works fine
> TB 3.0b2.Build2.de works fine
> TB 3.0b3.Build1.en-us slow html view on specific HTML-mails
> TB 3.0b3.Build1.de slow html view on specific HTML-mails

great. so now you should be able to narrow the range with nightly builds from spring 2009 between 3.0b2 and 3.0b3, roughly 2/25/2009 and 7/21/2009. take the "exact" middle date between the two, and in a few tests you should find a 1 day regression range.  you don't need to test both en-us and .de
2009-06-01-03-comm-1.9.1 works fine
2009-06-15-03-comm-1.9.1 works fine
2009-06-19-03-comm-1.9.1 works fine
2009-06-20-03-comm-1.9.1 with problem BINGO!
2009-06-21-03-comm-1.9.1 with problem
2009-06-23-07-comm-1.9.1 with problem
2009-06-30-03-comm-1.9.1 with problem
2009-07-01-03-comm-1.9.1 with problem
kudos to g.lamken!  We now have a regression range.  So requesting WANTED for v3.1.

nothing in the regression range jumps out at me as affecting message BODY. None of bugs I checked have regressions marked against them, or appear to be overtly performance related.
http://hg.mozilla.org/comm-central/pushloghtml?startdate=2009-06-19+03%3A00%3A00&enddate=2009-06-20+04%3A00%3A00
http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2009-06-19+03%3A00%3A00&enddate=2009-06-20+04%3A00%3A00
Status: UNCONFIRMED → NEW
blocking-thunderbird3.1: --- → ?
Ever confirmed: true
Keywords: regression
(In reply to comment #37)
> requesting WANTED for v3.1.

This is a pain point for users and a source of complaints. I know there are reports on gsfn and elsewhere but don't have any links ready. Anyone should feel free to collect them, as well as identify related confirmed or unconfirmed bugs.

Perhaps someone could also confirm g.lamken's regression range.
Whiteboard: [gs]
blocking-thunderbird3.1: ? → ---
Flags: wanted-thunderbird+
Huge tracemonkey checkin for mozilla-central might be eliminated as a source for the problem by toggling the following prefs in configedit:
javascript.options.jit.content
javascript.options.jit.chrome
(In reply to comment #39)
> Huge tracemonkey checkin for mozilla-central might be eliminated as a source
> for the problem by toggling the following prefs in configedit:
> javascript.options.jit.content
> javascript.options.jit.chrome

Is this something what I should check?
Actually I already changed this via about:config and restarted TB 2009-06-20-03-comm-1.9.1 and later the current version. It hasn't any positive effect.
(In reply to comment #40)
> Is this something what I should check?... It hasn't any positive
> effect.

Excellent - eliminated as a possible cause - and kudo's to Joe for the suggestion.  

I'm not sure which bugs are most likely to be related from a code POV, but eliminating linux only and such from the comm-central checkins leaves me with "Remove compact message header pane from the core code (bug 480623)". And for mozilla-central I have no clue. /boy if only we could reduce the list of possible checkins by half a day .../

Perhaps Magnus, standard8 or someone else can suggest a likely suspect from comment 37
Now a far fetched idea from a non developer, who noticed a similar behaviour in a total different program.
We are using a program, which displays a text file with linked images.
If there are UNC links used instead of Windows path, e.g. \\somewhereMachine\foopath\bla\etc instead C:\foopath\bla\etc, and the UNC is not available for some reason, than Windows takes some time and blocks the program until it seem for Windows to be clear that that Path doesn't work. 

In my user perspective TB3 behaves very similar. So my amateur question comes up:
Exists somewhere in the TB3 code or properties a UNC or similar Network access demand which sometimes couldn't be answered by Windows?
(In reply to comment #42)
> We are using a program, which displays a text file with linked images.
> If there are UNC links used instead of Windows path, e.g.
> \\somewhereMachine\foopath\bla\etc instead C:\foopath\bla\etc, and the UNC is
> not available for some reason, than Windows takes some time and blocks the
> program until it seem for Windows to be clear that that Path doesn't work. 
> In my user perspective TB3 behaves very similar. So my amateur question comes
> up:
> Exists somewhere in the TB3 code or properties a UNC or similar Network access
> demand which sometimes couldn't be answered by Windows?

It is similar to ordinal Network drive/file access. Time to detect timeout depends on network environment and setups of MS Windows. Check time that MS Windows detects timout of next commads at Command prompt.
  - NET VIEW \\somewhere
  - NET USE \\somewhereMachine\foopath X:
  - DIR \\somewhereMachine\foopath\bla
  - COPY \\somewhereMachine\foopath\bla\abc.txt C:\abc.txt
(In reply to comment #43)
> (In reply to comment #42)
> Check time that MS
> Windows detects timout of next commads at Command prompt.
>   - NET VIEW \\somewhere
>   - NET USE \\somewhereMachine\foopath X:
>   - DIR \\somewhereMachine\foopath\bla
>   - COPY \\somewhereMachine\foopath\bla\abc.txt C:\abc.txt

always few secs (2-5sec)

My example is a fuzzy hint and I can't decide whether it's worth to follow.

That test result is very different to the behaviour of the described problem.
That program I described as comparison has sections where linked images can be displayed.
If that program displays a file from a foreign system with unknown UNC path links, it behaves like this TB3 HTML-View problem. You have to wait a while. Than you can start normal work until you open another part with an broken link, where the time problem starts again.

If you have path with unknown MS drive letter, there is no time problem (besides the image can't be displayed either ;-) ).
(In reply to comment #44)
I forgot to mention. This UNC problem with my program (not TB) is reproducible on each system I've worked with. Means: 4 different big companies with different network systems, XP and Vista clients in Linux-samba, Windows 2003 and unknown network environments.
As far as I understand this, is, that it's a problem how a Windows client handles UNC path by default. Depending programs suffer. That means programs on Windows should avoid UNC in properties or other not manually changeable parts.
NetBIOS over TCP(NBT) uses broadcasting for name resolution, as it's NetBIOS.
See next documents.
> http://en.wikipedia.org/wiki/Server_Message_Block#Performance_issues
> http://en.wikipedia.org/wiki/NetBIOS
> http://technet.microsoft.com/en-us/library/bb727013.aspx
>   Chapter 11 - NetBIOS over TCP/IP
Because of "over TCP", the broadcasting can reach to far distant/far many  locations than traditional NetBIOS. I guess some settings such as timeout value, max hop counts, etc. exist, but I don't know well. Filtering of NetBIOS packect/SMP packet at appropriate point can be another workaround(ADSL modem usually filters NetBIOS/SMB packets by default.)

If \\server_name used in your intranet is known and usually-existent/active file server, defining IP address of server in LMHOSTS at each PC may be a workaround of too long timeout detection.

If drive letter is assigned to a shared resource of a file server(NET USE \\server\share_name X: is issued), name resolution is executed upon "NET USE" execution time. If server down happens, MS Win knows the down by timeout, and, once error occurs, MS Win probably immediately returns error(drive not exists) to subsequent request for the network drive letter.

If many HTML mails with many <img src="file://///non-existent-server-on-the-earth/non-existent-resource/dir/name.ext"> are frequently sent to you, I think one of next are practical circumvention of problem.
- Always set View/Message Body As/Plain Text
- Add non-existent-server-on-the-earth=192.168.0.x(your PC address) in LMHOSTS
Both are similar to self-protection from spam/phising mails.

By the way, in what environmet your problem does occur with HTML mail you attached to this bug(<img src="proper-cid:..."> in HTML)? In what environmet your problem doesn't occur with HTML mail you attached to this bug(<img src="proper-cid:..."> in HTML)?
I understand that it's quite a problem that I seem to be the only one (besides my colleagues) who can reproduce this BUG.

(In reply to comment #46)
> If many HTML mails with many <img
> src="file://///non-existent-server-on-the-earth/non-existent-resource/dir/name.ext">
> are frequently sent to you, I think one of next are practical circumvention of
> problem.
> - Always set View/Message Body As/Plain Text
> - Add non-existent-server-on-the-earth=192.168.0.x(your PC address) in LMHOSTS
> Both are similar to self-protection from spam/phising mails.

This is probably a misunderstanding. None of those mail in question are with something like <img src="file://///non-existent-server-on-the-earth/non-existent-resource/dir/name.ext"> (I know that this is just an example.)
Actually I'm not aware that any of my mails have a broken link.
Maybe some spam have such image sources but I don't open them.
I always view and sent mails as plain text, but can't force my job contacts (partner and customers) to do the same.
Especially some colleagues of mine have to read mails from partner and customer in HTML view, to get an faster overview of the content, especially if important textmarks are used which are composed in HTML.

Broken image links aren't the problem.
The long rendering time (or what ever) in HTML-view is.

> By the way, in what environmet your problem does occur with HTML mail you
> attached to this bug(<img src="proper-cid:..."> in HTML)? In what environmet
> your problem doesn't occur with HTML mail you attached to this bug(<img
> src="proper-cid:..."> in HTML)?
That versions where I deleted the whole attachment part with a text editor or that version after your suggestion to change <img to <ximg are displayed without images of course but loaded very fast.

I checked my old notebook environment. If I use the local copy of TB-'local folder' which usually is stored on a network folder (because of backup mechanism), than HTML-view is a bit faster but nothing in comparison to TB2.

Our network folder, where our mail folder are referring to, is assigned to a drive letter. We don't use UNC for TB or other user applications but know the use of it in other environments.

By the way, one colleague of mine just told me, he only uses plain text view, that he migrates back to TB2. He has problems when he switches fast between some mail.
Than by random TB3 takes a break and he has to wait too much. He can't reproduces this with specific mails. It just happens enough to get on his nerves. This didn't happen with TB2.
I'm concerned that bug 536873 doesn't seem to be making progress, and http://gsfn.us/t/o85n depends on that bug (with 113 watchers, although it's unclear how many of that 113 really have a thunderbird bug).

I'm not convinced this bug is clearly a big help to http://gsfn.us/t/o85n, but hooking it up anyway because it is at least making headway
URL: http://gsfn.us/t/o85n
Wayne Mery pointed this bug and bug 542747 in bug 536873 comment #9. Request to clean up %temp% and Disk Cache is to rule out phenomenon like bug 542747.
> Bug 536873 TB3.0 slow, takes a long time to load HTML messages with View Message Body as Original HTML,
> sits at "Loading Message ..." or presents blank screen
> Bug 542747 TDB Eating up the disk when try to open a mail with many images

Is there any common condition in environment of bug reportes of these bugs?
Problem like next is involved?
> bug 543082 slow Thunderbird when opening an email while npOGPPlugin extension
>            from ogplanet.com is installed (Game Launcher)

(In reply to comment #6)
> Outlook Express(probably .pst is placed on local HDD) 
>  (2) 000:00:00.575
> Tb, held on local HDD, original, with <img>
>  (4)a 000:02:12.645 original mail
>  (4)b 000:02:28.538 anonymized mail you got
> Tb, held on local HDD, without <img>
>  (8) 000:00:00.555

(4)b can be replaced by test case you attached to this bug.
g.lamken@web.de, can you get Process Monitor log for (case-1) test HTML in local mail folder, (case-2) test HTML without <img> in local mail folder?

> Process Monitor :
> http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
(1) Start Process Monitor,
    Filter: If process is thunderbird.exe, Include
    Stop Capture, Clear Display
(3) Clear Display, Start Capture,
    Execute (case-1). View test mail with <img>    => it takes long
    Stop Capture, Save monitor log(to .csv file)
(4) Clear Display, Start Capture,
    Execute (case-2). View test mail without <img> => no problem 
    Stop Capture, Save monitor log(to .csv file)
What is difference of process monitor log between (case-1) and (case-2)?
Justin, checkin of bug 469443 - Form Manager Storage should be a JavaScript-based component - is in the regression range for this bug [1]. Is there an easy way to eliminate it as a possible source of this regression? 

g.lamken, can you profile as suggested by wada?

Bug 495959 is a possibility.  Many checkins were tests. Eliminating others, like ARIA, doesn't leave much. And tracemonkey checkin was eliminated by testing with JIT disabled.

[1] http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2009-06-19+03%3A00%3A00&enddate=2009-06-20+04%3A00%3A00
(In reply to comment #49)
> > Process Monitor :
> > http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
> (1) Start Process Monitor,
>     Filter: If process is thunderbird.exe, Include
>     Stop Capture, Clear Display
> (3) Clear Display, Start Capture,
Which of following 'show options' are wanted?
- Show Registry Activity
- Show File System Activity
- Show Network Activity
- Show Process and Thread Activity
- Show Profiling Events
At least the last one is running continuously without helpful information.
Sorry for ambigous request. Check file access by Tb first, please.
> - Show File System Activity
At which steps (case-1) takes longer or (case-1) accesses many files than (case-2)?
(In reply to comment #52)
I've sent WADA requested log files.
It makes no sense that I'm blind fishing for details and probably missing exact that detail, he is looking for.
Even there are no astonishing secrets about my environment it would be bad style and weak security sensibility to publish this the forum.
I did this logging two times to make sure, that I really captured the right moment.
It's tricky not to capture "mouse over" events, which are not helpful for this comparison.
Sorry,
that I always forget important information.
That testing mails are in folder called 'aaaa'.
(In reply to comment #53)
> It's tricky not to capture "mouse over" events, which are not helpful for this comparison.

As test of mail viewing, first log of "read mail data(roughly 4KB+2KB) at ...\aaaa" can be used as first step of each test. So, comparison of time between the log and time when the mai is displayed, and comparson of logs after it till time of "end of display", has meaning as first analysis step, although log of file access only is not sufficient data for diagnosis.
(In reply to comment #55)
Unfortunately I am travelling and can't test this until Friday
I received file access log data via email. 

(Quick log data check result)
Following is first log for read of mail data from Tb's mail folder file.
  Data size = 4096 + 4096 + 1897 bytes.
(WithImg.CSV, Log for whole mail data read)
> "08:59:57,9469081", ,"ReadFile",
>    "X:\MozillaProfiles\Thunderbird\ ... \Mail\Local Folders\aaaa",
>           "SUCCESS","Offset: 6.987, Length: 4.096","0.0000720"
> "08:59:57,9472289", , "ReadFile",
>    "X:\MozillaProfiles\Thunderbird\ ... \Mail\Local Folders\aaaa",
>           "SUCCESS","Offset: 11.083, Length: 4.096","0.0000384"
> "08:59:57,9472913", ,"ReadFile",
>    "X:\MozillaProfiles\Thunderbird\ ... \Mail\Local Folders\aaaa",
>           "SUCCESS","Offset: 15.179, Length: 1.897","0.0000231"
> "08:59:57,9473430", ,"CloseFile",
>    "X:\MozillaProfiles\Thunderbird\ ... \Mail\Local Folders\aaaa",
>           "SUCCESS","","0.0000863"

Above set of log lines was seen 8 times.
(1) "08:59:57,9469081", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0000720"
(2) "09:00:00,2508265", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0001136"
(3) "09:00:02,3478872", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0000493"
(4) "09:00:04,5135987", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0020552"
(5) "09:00:06,6813114", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0000538"
(6) "09:00:09,0905262", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0024256"
(7) "09:00:11,3588574", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0001063"
(8) "09:00:13,6110473", ,"ReadFile","X:\Moz ... \Local Folders\aaaa",
                "SUCCESS","Offset: 6.987, Length: 4.096","0.0001024"

It indicates that Tb 3.0 reads whole mail data for each <img src="cid:...">. I think that this incident itself is not cause of the problem, although this can make it worse if network file and big HTML. It looks that rendering of each small image takes 2-3 seconds in your environment. As log for file write is not seen, base64 decoding and rendering of image looks to be executed in memory.

Very many logs for directory/file for mail folder and C:\Windows\CSC\... are seen between above log. It is probably log of periodical mail folder access(msgpurge, gloda, ...), and I think it's irrelevant to performance issue of this bug. These many logs are merely written many times while rendering, because img rendering takes long.

Next problem determination step:
- Process Monitor(or filemon): File access of ...\aaaa only.
    (to catch start of processing of an <img src="cid:..."> in HTML)
- Process Monitor(or filemon): Other logs
    (to catch events/processes while Tb is rendering image)
- NSPR log: all:5
    (start of mail viewing can be tracked by log for "Internal Load")

g.lamken@web.de(bug opener):

Can you check with Tb's profile on local HDD for testing of this bug/mail folder file on local HDD, instead of Tb's profile on file server/mail folder of "Local Folders" on file server?
I think this bug is never for problem due to network file and/or MS Win's Offile File(CSC). I think this bug is irrelevant to Tb3's bug in file writing(no buffering is used in write), because Process Monitor log of file write which looks relevant to HTML mail viewing is not found in log by my test and log you sent me.
However, simplest test is better for diagnosis.

Can you test with identical mail data with HTML mail you attached to this bug, and with modified version of the test mail(<img => <xmg, same structure, same length) if check of "no image rendering" case at same time is required?

Can you check "Order Received" column value of test mails for ease of log analysis when test will be executed next time? ("Order Received" column value is offset in mail folder file if local mail folder, and it appears in file access log, "Internal Load" log by NSPR logging etc.)
Local profile/local mail folder(Bug-Test.sbd/bug-545126).
HTML mail attached to this bug (7 <img src="cid:...">).
Offset=0 ==> ?number=0 in log
Checked with Tb 3.0.3, MS Win-XP SP3 on Intel Core2 Duo, 1.66GHz, 1GB RAM.

My NSPR log data with all:5 is consistent with Process Monitor(or Filemon) log in your environment.
In both tests :   1 whole mail data read for mail view
                + 7 whole mail data read (for each <img>)
Difference is:
  Your test : Read is requested each 2-3 seconds.
  My test   : Read is requested each 15 miliseconds.

> 00000000 11:32:51.781 [4844] 0[182a140]: DOCSHELL 31e4000 InternalLoad mailbox:///C|/wada/@@@/Mail1/Bug-Test.sbd/bug-545126?number=0 
(First read of whole mail data by mail view)
> 00000001 11:32:51.781 [4844] 4572[3caf780]: read -> 4096 
> 00000002 11:32:51.781 [4844] 4572[3caf780]: read -> 4096 
> 00000003 11:32:51.781 [4844] 4572[3caf780]: read -> 1897 

>(many logs like next after plugin load. logs for update of pluginreg.dat. one per a line due to bug 539389)
> 00000105 11:32:51.828 [4844] 0[182a140]: write -> 29 
> 00000669 11:32:51.890 [4844] 0[182a140]: write -> 34 
>
> 00000675 11:32:51.890 [4844] 0[182a140]: DOCUMENT 37bc800 created 
> 00000676 11:32:51.890 [4844] 0[182a140]: NODEINFOMANAGER 4a87e80 created 
> 00000677 11:32:51.890 [4844] 0[182a140]: NODEINFOMANAGER 4a87e80 Init document=37bc800 
> 00000678 11:32:51.890 [4844] 0[182a140]: DOCUMENT 37bc800 StartDocumentLoad mailbox:///C|/wada/@@@/Mail1/Bug-Test.sbd/bug-545126?number=0 
> 00000679 11:32:51.890 [4844] 0[182a140]: DOCUMENT 37bc800 ResetToURI mailbox:///C|/wada/@@@/Mail1/Bug-Test.sbd/bug-545126?number=0 
> 00000680 11:32:51.890 [4844] 0[182a140]: DOCSHELL 31e4000 SetCurrentURI mailbox:///C|/wada/@@@/Mail1/Bug-Test.sbd/bug-545126?number=0 

> 00000683 11:32:51.921 [4844] 0[182a140]: Mailbox Done 

(read of whole mail data for img #1, probably)
> 00000687 11:32:51.937 [4844] 4572[3caf780]: read -> 4096 
> 00000688 11:32:51.937 [4844] 4572[3caf780]: read -> 4096 
> 00000689 11:32:51.937 [4844] 4572[3caf780]: read -> 1897 
(read of whole mail data for img #2, probably)
> 00000690 11:32:51.953 [4844] 4572[3caf780]: read -> 4096 
> 00000691 11:32:51.953 [4844] 4572[3caf780]: read -> 4096 
> 00000692 11:32:51.953 [4844] 4572[3caf780]: read -> 1897 
(read of whole mail data for img #3, probably)
> 00000693 11:32:51.968 [4844] 4572[3caf780]: read -> 4096 
> 00000694 11:32:51.968 [4844] 4572[3caf780]: read -> 4096 
> 00000695 11:32:51.968 [4844] 4572[3caf780]: read -> 1897 
(read of whole mail data for img #4, probably)
> 00000696 11:32:51.968 [4844] 4572[3caf780]: read -> 4096 
> 00000697 11:32:51.968 [4844] 4572[3caf780]: read -> 4096 
> 00000698 11:32:51.968 [4844] 4572[3caf780]: read -> 1897 
(read of whole mail data for img #5, probably)
> 00000699 11:32:51.984 [4844] 4572[3caf780]: read -> 4096 
> 00000700 11:32:51.984 [4844] 4572[3caf780]: read -> 4096 
> 00000701 11:32:51.984 [4844] 4572[3caf780]: read -> 1897 
(read of whole mail data for img #6, probably)
> 00000702 11:32:52.000 [4844] 4572[3caf780]: read -> 4096 
> 00000703 11:32:52.000 [4844] 4572[3caf780]: read -> 4096 
> 00000704 11:32:52.000 [4844] 4572[3caf780]: read -> 1897 
(read of whole mail data for img #7, probably)
> 00000705 11:32:52.015 [4844] 4572[3caf780]: read -> 4096 
> 00000706 11:32:52.015 [4844] 4572[3caf780]: read -> 4096 
> 00000707 11:32:52.015 [4844] 4572[3caf780]: read -> 1897 

(normal end of display, probably)
> 00000713 11:32:52.031 [4844] 0[182a140]: Mailbox Done 
> 00000714 11:32:52.031 [4844] 0[182a140]: Mailbox Done 
> 00000715 11:32:52.031 [4844] 0[182a140]: Mailbox Done 
> 00000716 11:32:52.031 [4844] 0[182a140]: Mailbox Done 
> 00000717 11:32:52.031 [4844] 0[182a140]: Mailbox Done 
> 00000718 11:32:52.031 [4844] 0[182a140]: Mailbox Done 
> 00000719 11:32:52.031 [4844] 0[182a140]: Mailbox Done
g.lamken@web.de, according to your comment #6, mail display took more than 2 minutes.
> (4)a 000:02:12.645 original mail
> (4)b 000:02:28.538 anonymized mail you got
But, according to Process Monitor(or Filemon) log you sent to me, it looks that mail display took less than 16 sec.
What is difference between your test of comment #6 and your test to get file access log?
I looks that Tb3.0 quiries all directories under Tb's \Mail and \News directory upon each open of a mail folder file.
It was also seen in file access log sent me via mail from bug opener.
  Quiry log for all of next directory was seen in your log.
    X:\...\Mail\Local Folders\*.*,
    X:\...\Mail\mail.paradatec.de\*.*
    X:\...\Mail\Feeds\*.*
    X:\...\News\*

If the directories/files under the directory are MS Win's Offline Files(CSC), next acess happens for each query of a directory. It looks to take 20msec in bug opener's envirinment.
"08:59:58,9814619","thunderbird.exe","2988","QueryDirectory","X:\MozillaProfiles","SUCCESS","Filter: MozillaProfiles, 1: MozillaProfiles","0.0005907"
> "08:59:58,9820876","thunderbird.exe","2988","CloseFile","X:\","SUCCESS","","0.0000141"
> "08:59:58,9827321","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000116"
> "08:59:58,9832334","thunderbird.exe","2988","CreateFile","X:\MozillaProfiles","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened","0.0017500"
> "08:59:58,9850097","thunderbird.exe","2988","QueryDirectory","X:\MozillaProfiles\Thunderbird","SUCCESS","Filter: Thunderbird, 1: Thunderbird","0.0006194"
> "08:59:58,9856581","thunderbird.exe","2988","CloseFile","X:\MozillaProfiles","SUCCESS","","0.0000125"
> "08:59:58,9866015","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000116"
> "08:59:58,9872168","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000094"
> "08:59:58,9891441","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000177"
> "08:59:58,9904788","thunderbird.exe","2988","CreateFile","X:\MozillaProfiles\Thunderbird\Guido\Mail","SUCCESS","Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, AllocationSize: n/a, OpenResult: Opened","0.0021278"
> "08:59:58,9932290","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000120"
> "08:59:58,9949800","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000114"
> "08:59:58,9963660","thunderbird.exe","2988","QueryDirectory","X:\MozillaProfiles\Thunderbird\Guido\Mail\mail.paradatec.de","SUCCESS","Filter: mail.paradatec.de, 1: mail.paradatec.de","0.0004428"
> "08:59:58,9968733","thunderbird.exe","2988","CloseFile","X:\MozillaProfiles\Thunderbird\Guido\Mail","SUCCESS","","0.0000245"
> "08:59:58,9977352","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000114"
> "08:59:58,9984460","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000095"
> "08:59:59,0003239","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000145"
> "08:59:59,0021033","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000124"
> "08:59:59,0032193","thunderbird.exe","2988","QueryOpen","X:\MozillaProfiles\Thunderbird\Guido\Mail\mail.paradatec.de\Junk","FAST IO DISALLOWED","","0.0000039"
> "08:59:59,0041864","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000099"
> "08:59:59,0050611","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000092"
> "08:59:59,0066998","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000100"
> "08:59:59,0085634","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000101"
> "08:59:59,0097223","thunderbird.exe","2988","CreateFile","X:\MozillaProfiles\Thunderbird\Guido\Mail\mail.paradatec.de\Junk","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened","0.0014070"
> "08:59:59,0117230","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000120"
> "08:59:59,0134404","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000105"
> "08:59:59,0152018","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000101"
> "08:59:59,0162162","thunderbird.exe","2988","QueryBasicInformationFile","X:\MozillaProfiles\Thunderbird\Guido\Mail\mail.paradatec.de\Junk","FAST IO DISALLOWED","","0.0000041"
> "08:59:59,0162324","thunderbird.exe","2988","QueryBasicInformationFile","X:\MozillaProfiles\Thunderbird\Guido\Mail\mail.paradatec.de\Junk","SUCCESS","CreationTime: 17.3.2010 16:51:12, LastAccessTime: 17.3.2010 16:51:12, LastWriteTime: 24.3.2010 08:10:41, ChangeTime: 24.3.2010 08:10:41, FileAttributes: A","0.0005091"
> "08:59:59,0167584","thunderbird.exe","2988","CloseFile","X:\MozillaProfiles\Thunderbird\Guido\Mail\mail.paradatec.de\Junk","SUCCESS","","0.0000201"
> "08:59:59,0174439","thunderbird.exe","2988","ReadFile","C:\WINDOWS\CSC\00000001","SUCCESS","Offset: 0, Length: 64","0.0000114"

If this is cause of performance issue of this bug, these exess file I/O's were possibly produced by patch for Mal&News Core bug 497680 which is listed in change log corresponds to regression range.
>(A part of change by bug 497680)
> +  nsCOMPtr<nsIArray> folderArray;
> +  accountManager->GetAllFolders(getter_AddRefs(folderArray));
> +
> +  PRUint32 count;
> +  rv = folderArray->GetLength(&count);
>    NS_ENSURE_SUCCESS(rv, rv);
> +  for (PRUint32 i = 0; i < count; i++)
> +  {
> +    nsCOMPtr<nsIMsgFolder> folder(do_QueryElementAt(folderArray, i, &rv));
> +    NS_ENSURE_SUCCESS(rv, rv);
> +    nsCOMPtr<nsILocalFile> folderPath;
> +    rv = folder->GetFilePath(getter_AddRefs(folderPath));
> +    NS_ENSURE_SUCCESS(rv, rv);
> 
> +    // Check if we're equal
> +    PRBool isEqual;
> +    rv = folderPath->Equals(aLocalPath, &isEqual);
> +    NS_ENSURE_SUCCESS(rv, rv);
> +    if (isEqual)
> +      return folder->GetURI(mailboxUri);
>   }

And, Tb 3.0.3 opens mail folder file for each <img src="cid:...">, situation becomes far worse if many <img src="cid:..."> is written in HTML...
sid, your thoughts on comment 60 and bug 497680?
(In reply to comment #59)
> g.lamken@web.de, according to your comment #6, mail display took more than 2
> minutes.
> > (4)a 000:02:12.645 original mail
> > (4)b 000:02:28.538 anonymized mail you got
> But, according to Process Monitor(or Filemon) log you sent to me, it looks that
> mail display took less than 16 sec.
> What is difference between your test of comment #6 and your test to get file
> access log?

That logs are exactly only results of one action: "display of one email by set mouse focus (click) on that specific email entry in email list".
I started capture in procmon, clicked email in question, waited until email is displayed and stopped capture.
Without attached images (cut with text editor) you have about 300 (plus something) file access actions.
With images you have about 1300 (plus xx) actions.
Emails are displayed when these actions are finished. Therefore I think you have to take these actions in total (minus parallel actions if this is possible).
Although this test is done in -safe-mode and automatic reload of accounts are disabled, there are always these file access which for me seems to be lookups for Add-Ons and Windows cache actions.
There are less lookups without images.

What's about those "FAST IO DISALLOWED"?

I admit that these anonymized email are displayed faster now than it was when I started these tests. That old test was in that dummy account which is deleted meanwhile. Know these emails are under folder 'aaaa' in 'Local Folders' and take about 1 min instead of 2 to be displayed, like the title of this thread is.

The original email in the original folder, where the anonymized is taken from and other emails from same source take more time than the anonymized one.
On my old notebook with less power than the desktop they are displayed slightly faster, what seems to me, that it's depending from environment, and maybe from the age of environment (changes in registry, install and uncompleted uninstall of file-viewer links).
Quick summary of my observations:
(1) If MS Win's Offline Files(CSC) is used, [ { 20msec(in bug opener's
    environment) } * { number of directories under \Mail, \News } ]
    is added to "time required to open a mail folder file".
(2) Mail folder file is opened and whole mail data is read
    { 1 + (number of <img src="cid:..."> in HTML) } times
    upon viewing of HTML mail with inline images.

g.lamken@web.de, can above summary explain difference between your two tests(2min vs. 16sec)?
Correction. wrong: { number of directories under \Mail, \News }. It should be;
  { number of mail folder files under \Mail, \News }
(In reply to comment #60)
> I looks that Tb3.0 quiries all directories under Tb's \Mail and \News directory
> upon each open of a mail folder file.
Not all possible folders visible in the log. I remember subfolder are also noted. Therefore there are much more folder possible to be shown in logs.
'zu erledigen' is a virtual folder witch looks for decent tagged mail in other folder. But the rule for this is not identical with the entries in logs.

(In reply to comment #63)
Sorry, have to leave train. Check later.
(In reply to comment #65)
> (In reply to comment #60)
> > I looks that Tb3.0 quiries all directories under Tb's \Mail and \News directory
> > upon each open of a mail folder file.
> Not all possible folders visible in the log. I remember subfolder are also
> noted. Therefore there are much more folder possible to be shown in logs.
> 'zu erledigen' is a virtual folder witch looks for decent tagged mail in other
> folder. But the rule for this is not identical with the entries in logs.

Sorry, I didn't want to mislead you.
My logs are for me available again now. Quiries of folders are always first stage, means no subfolder.
(In reply to comment #66)

Hello. I'm having the same problem and thought about adding my time into helping to fix this thing. I've done tests and found a way to have the problem down to nearly zero: reduce greatly the number of folders in TB. Then mails with HTML images display quickly.

Using procmon, it shows that TB does an incredibly long attempt to "create file" (and other file operations) in each and every mail folder, but doesn't do that when images do not contain images (i.e., plain text). A capture example with procmon shows (on my PC) 1255 lines of CreateFile operation when opening text files and 17515 lines of CreateFile when opening HTML mail with images.

Example lines are like this:

"1:16:27.2600797 PM","thunderbird.exe","2828","CreateFile","C:\Users\%Username%\Documents\Mail\Thunderbird\Mail\Local Folders\Inbox","SUCCESS","Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened"

I'm planning to retest with both versions 2009-06-19-03-comm-1.9.1 (works fine)
2009-06-20-03-comm-1.9.1 (with problem BINGO!) and see the difference.

Anyway, it is clear why not everybody's having this problem: unless you have a very complex mail folder structure you're not likely to see the problem or it won't probably bother you.
(In reply to comment #66)

Ok, here's the second part of the problem: TB searches each mail folder when opening HTML messages with images, but does so in alphabetical order. Now, personally my mail archive is organized by years. So I have folders named 1997, 1998, 1999 and so on. They alphabetically are before folders such as Inbox. The quick fix for this problem: I've created a folder named Z-Archive. Moved all my older messages exactly as they were and with the same folder structure into Z-Archive and...presto!

It works: no more delays. ProcMonitor shows TB only opening the folder with the message in question and it takes about 0.5s (before it was about 15-20s).

Now, regarding the reasons, I can confirm the problem doesn't happen with TB 3b3 dated 03-19 but does with the one dated 03-20 (and all versions after that, including 3.1 beta). Also, I've tried a "custom" V 03-20 of which only thunderbird.exe was changed with the one of 03-19 and the problem doesn't occur, so it's related to thunderbird.exe and no other files.

Does this ring any bell? Was something changed between 03-19 and 03-20 that causes searching alphabetically through all folders when an HTML mail with an image is displayed?
Ivan, so you seen a behavior change at both breaks?  2009-03-19/2009-03-20 *and* 2009-06-19/2009-06-20?

* comm-central http://hg.mozilla.org/comm-central/pushloghtml?startdate=2009-03-19+03%3A00%3A00&enddate=2009-03-20+04%3A00%3A00
* moz-central http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2009-03-19+03%3A00%3A00&enddate=2009-03-20+04%3A00%3A00

what stands out from me from gecko changes is  Bug 422526 -  implement localStorage
(In reply to comment #69)

Wayne, definitely the problem started with 3.0b3 of 03-20. I went through many revision changes and I agree 100% with g.lamken.

Let me prepare another test (since now it's no use with the folder called Z-archive. I will try with 2 folder: Z-archive and 0-archive and move messages from one to the other)

BTW, not really an expert in winApp here, but what about  Bug 497242 -  [FIX]getElementsByName should not find non-HTML nodes  

Regards,
I can believe that my patch caused the regression here -- using a debugger + procmon, I found that each equals comparison causes a call to GetShortPathNameW at <http://mxr.mozilla.org/comm-central/source/mozilla/xpcom/io/nsLocalFileWin.cpp#3053>, which in turn opens the file and whatever else is mentioned in comment 60. Since this is essentially O(N) random disk I/O in the total number of folders you have, it could potentially hurt a lot.

Since for correctness purposes it seems like we need to compare every folder, what we seem to need here is a map from folder paths to folders, probably implemented as a hash table. nsIFolderLookupService maybe? David, any other ideas to reduce the number of folders to compare here?
(In reply to comment #69)
(In reply to comment #71)

Hi Wayne , hi Siddharth.

Ok, I've got the proof: unmistakably the problem started on 03-20. ProcMonitor confirms that 03-20 is actually attempting to scan each and every folder in alphabetical order. Clearly, if someone has bunch of stuff (read GB) in folders name with years, it's a complete disaster.

I can send you the .PML files. Since it contains a bit of personal info (nothing major), I'd rather not post them online, so if you need them, please let me know.

Regards,
very cool diagnosis. couple questions:

1. I've got about 100 folders, most of them subfolders, but I don't really see terrible effects. Should that be enough to to see the problem?  I don't see a big hit, but perhaps because of my machine speed plus my it has 10krpm drive. Does anyone have a ubergood badass testcase that will make my machine puke? 
2. Is the impact of these regressions limited to when one displays a message? Or might it also be felt at other times, like picking folder names from menus in various dialogs and such?
3. related to 2 above, assuming someone who is going through and viewing lots of messages (and they are all HTML, blah, blah) how badly might this impact the performance of the machine overall, and background operations like gloda?
p.s. bear in mind some people have hundreds and even thousands of mail subfolders even on oldish machine
sid0, this would be a good reason to revive the folder lookup service...
Wayne:
If you go back and read WADA's very nice analysis in comment 57
This seems to just exacerbate (due to networking I gather) a basic weakness
in rendering embedded images.
"It indicates that Tb 3.0 reads whole mail data for each <img src="cid:...">. I
think that this incident itself is not cause of the problem, although this can
make it worse if network file and big HTML."
In other words, this search down through the file structure is repeated every
time an image CID is encountered.(due to the bug)

The fact that the message must be located in the file structure and completely
read again, each time a CID is found must be expensive, even without the conditions of this bug.
(I often see 12-14 embedded images in mail and newsgroup posts)
Hi Wayne,

1) I've got... well, 816 folders!!! Ok, I know about "why having so many ?",
etc. etc. but I have quite a lot of stuff that I need to keep handy and really
well organized. So daily I might look into stuff that is from the 2002 or 2003
folder (and the many subfolders)

2) As far as I can tell, display of folders is not affected at all. It's only
when you display/open a message with an image. Now, if you have the message
pane open, it might seems that in some cases folder display takes a long time,
but in reality it doesn't (just close the message pane and TB is fast).
However, TB is also slow when opening attachments in a HTML mail with graphics
images in it. For example, the typical corporate E-mail, with the fancy logo
and the attachment. Open the attachment and it takes a long time. Using the
Z-Archive naming approach fixes this problem, as well.

3) From my point of view, if someone has this problem it's a "drop TB" kind of
problem. On a single-core machine and with more folders (I saw some time ago
that purging empty folders in older archives had a positive effect -now I know
why-) it took 30-45s or more to display a message. When you're looking at bunch
of corporate E-mail and each has the small .JPG logo, it makes you crazy how
much time you're wasting... Right now I can live with the folder renaming
because at least in inbox is fast. But anything inside the "Z-Archive" folder
is still slow (consider 15s to display a message in a dual core machine with
4GB ram)
Yeah, what I'm after in comment 73 item 1 is I'd like to replicate to see the behavior myself, but what I've tested so far is pretty minimal impact. 

only one unclear point (because imap is mentioned only once), are messages in imap folder structure equally affected?
Wayne, one suggestion: can you create an empty folder in TB, and name it, say, 0000 then create maybe 10 empty folders in it and then use Windows to copy-paste to create bunch of copies of these 10 folders in folder 0000. It shouldn't take much time to create a folder structure with a thousand or so folders. Then you can try the impact of this problem by opening an HTML mail with an image in it maybe in Inbox (or any other folder alphabetically after 0000).

As per me, all are local folders. I'm not using IMAP so I can't say regarding IMAP folders.

Regards,
Simplified test cases based on example mail attached by bug opener.
 Case-00: multpart/mixed, 4 attached image/jpeg    => # of mail read == 1
 Case-01: multpart/related,  4 <img src="cid:...."> => # mail read == 1 +  4
 Case-02: multpart/related, 40 <img src="cid:...."> => # mail read == 1 + 41

Check result with Tb 3.0.3 on MS Win-XP SP3, with local profile/mail folder.
 Case-01: it took roughly  1 second to display mail.
 Case-02: it took roughly 10 second to display mail.

          In my environment, it took roughly 10 seconds to display.
(In reply to comment #78)
> I'd like to replicate to see the behavior myself, (snip)

Wayne Mery, if local profile/local mail folder, check with case-02, with getting Process Monitor log. Because (a), (b), and (c) occurs, performance issue is exposed even with local profile/local mail folder.
 (a) "open of mail folder file" * 41 times.
     (Existent/still-remail problem in <img src="cid:..."> processing)
 (b) "getting file name of all mail folder files upon open"  * 41 times
     (After change by bug 497680)
 (c) "read of whole mail data(88KB)" * 41 times
     (Existent/still-remain problem in <img src="cid:..."> processing)

Quick summry of this bug and Bug 536873.
- Before change by bug 497680, performance issue by (a)+(c) is exposed
  only when very many inline <img> and huge HTML.
- After change by bug 497680, time for (b) is added to time for (a)+(c).
  Then performance problem is exposed with medium number of <img>
  and with medium mail data size(Bug 536873).
  If MS Win's Offline Files(CSC) is used, time for (b) increases very much,
  thus performance problem is exposed with small number of <img> and with
  small mail data size(this bug).
(In reply to comment #65)
> (In reply to comment #63)
> Sorry, have to leave train. Check later.
WADA told me you have enough material and I'm released from further tests.
So some open questions where I was addressed needn't be answered.

My best wishes for you and your work
(In reply to comment #72)
> (In reply to comment #69)
> (In reply to comment #71)
> 
> Hi Wayne , hi Siddharth.
> 
> Ok, I've got the proof: unmistakably the problem started on 03-20. ProcMonitor
> confirms that 03-20 is actually attempting to scan each and every folder in
> alphabetical order. Clearly, if someone has bunch of stuff (read GB) in folders
> name with years, it's a complete disaster.
> 
> I can send you the .PML files. Since it contains a bit of personal info
> (nothing major), I'd rather not post them online, so if you need them, please
> let me know.

OK, I'm really confused now. Did the problem start on 20 *March* 2009 or on 20 *June* 2009?

Could you test the following builds:
2009-03-19
2009-03-20
2009-06-19
2009-06-20

and see what sort of performance you get?
(In reply to comment #85)

Siddarth, sorry. My stupidity. It was quite late last night and somehow 06-20 became 03-20 and from there it was all downhill.

I confirm the problem started on 06-20. I was testing the builds of 06-19 and 06-20. 06-19 works flawlessly and 06-20 does not.

Again, sorry about it.
FYI.
Performance problem of Tb 3.0.3 was not observed with IMAP folder using Case-02(40 <img>, 88KB), although bug 552806 occurs if IMAP folder of "offline use=off".
I was thinking of using a nsILocalFile's persistentDescriptor as the key, but if I remember correctly, the traditional Mac persistentDescriptor (Mac alias) only works for files that actually exist. So this won't work for folders that aren't offline.

Bug 506814 switched persistentDescriptor to using paths, but that's only on 1.9.3. I guess on Mac it's possible to implement a fallback map with path names if the folder isn't offline...
(In reply to comment #88)

As I wrote in comment #88, no performanse problem was observed with IMAP folder, for both "offline use=off" IMAP folder("mail data" is placed in file held in Disk Cache) and "offline use=on" IMAP folder("offlin-store file of IMAP" is similar to mail folder file of local mail folder, from "HTML mail rendering" view).

(Q1) Why does 'try to read of whole mail data to process <img src="cid:...">' invoke "mail folder file open process"(which invokes "directory access for all files under Tb's directories which are to hold Tb's mail folder files" after change of bug 497680)?
(Q2) Even if very expensive "file path comparison with all Tb's mail folder files after change of bug 497680" is invoked upon mail folder file open, I think that "this bug's performance issue due to change of bug 497680" won't occur, if "open of mail folder file" will not be invoked for each processing of <img src=cid:...> in HTML mail, even if "processing of <img src=cid:...>" invokes "read of whole mail data".
Siddharth Agarwal, what do you think?
(In reply to comment #89)
> (In reply to comment #88)
> 
> As I wrote in comment #88, no performanse problem was observed with IMAP
> folder, for both "offline use=off" IMAP folder("mail data" is placed in file
> held in Disk Cache) and "offline use=on" IMAP folder("offlin-store file of
> IMAP" is similar to mail folder file of local mail folder, from "HTML mail
> rendering" view).

IMAP has a completely different code path, so that doesn't apply here. IMAP seems to be able to create the URI without needing to look for which folder the file is in.

> 
> (Q1) Why does 'try to read of whole mail data to process <img src="cid:...">'
> invoke "mail folder file open process"(which invokes "directory access for all
> files under Tb's directories which are to hold Tb's mail folder files" after
> change of bug 497680)?

That's because for each such image with a cid, docshell code pokes libmime code to convert streams, etc. libmime is interested in getting the folder's charset. To get the folder's charset you need to know the folder, and to know the folder you need to know the URI.

So the call stack looks something like

mail.dll!nsMailboxUrl::GetUri
mail.dll!nsMailboxUrl::GetFolder
mail.dll!nsMailboxUrl::GetFolderCharsetOverride
mail.dll!bridge_new_new_uri
mail.dll!nsStreamConverter::SetStreamURI
mail.dll!nsStreamConverter::Init
mail.dll!nsStreamConverter::AsyncConvertData
necko.dll!nsStreamConverterService::AsyncConvertData
...

Now the trouble is that this is a necko-created URL, so mURI isn't set (note that mURI is a mailnews URI -- mailbox-message:// -- and is different from the spec of a necko-created URL, which in this case is mailbox://). So the code tries to create a mailnews URI out of the file path, and for that it needs to know which folder the file is from. <http://mxr.mozilla.org/comm-central/source/mailnews/local/src/nsMailboxUrl.cpp#186>

> (Q2) Even if very expensive "file path comparison with all Tb's mail folder
> files after change of bug 497680" is invoked upon mail folder file open, I
> think that "this bug's performance issue due to change of bug 497680" won't
> occur, if "open of mail folder file" will not be invoked for each processing of
> <img src=cid:...> in HTML mail, even if "processing of <img src=cid:...>"
> invokes "read of whole mail data".

Clearly that's another way this could be solved. However I really doubt a mailbox-message URI can be created solely out of a mailbox URL, because a mailbox URL looks like "mailbox:///C|/.../Thunderbird/Profiles/bejnf55f.Test1/Mail/Local%20Folders/Trash?number=0&part=1.3&filename=Teil%201.3.gif" -- i.e. it doesn't contain information about which folder we're in, only the path on the hard drive.
(In reply to comment #90)

Thanks for detailed explation f problem.

> Now the trouble is that this is a necko-created URL, so mURI isn't set (note
> that mURI is a mailnews URI -- mailbox-message:// -- and is different from the
> spec of a necko-created URL, which in this case is mailbox://).

I couldn't imagine that necko is relevant to this bug which is relevant only to already held mail in "local mail folder" and "local mail folder file"...
(Q3) Why necko is relevant in this bug's case?
     necko-created URL is required to download mail from POP3 server,
     and the internal necko-created URL for mail folder is commonly used
     in both networking-POP3 and locally saved mail handling?
Woops. It should be;
  Thanks for detailed explanation of problem.
(In reply to comment #91)
> I couldn't imagine that necko is relevant to this bug which is relevant only to
> already held mail in "local mail folder" and "local mail folder file"...
> (Q3) Why necko is relevant in this bug's case?

That's because when you display one of these messages, the docshell gets necko to create URLs for any sub-parts of the message -- specifically, <https://developer.mozilla.org/en/nsIIOService#newURI%28%29>. I call these URLs "necko-created". Don't get me wrong here -- I'm not saying that necko is in any way to blame.
(In reply to comment #93)
> I call these URLs "necko-created". Don't get me wrong here -- I'm not saying that necko is in any way to blame.

Oh, sorry for my bad question. I wanted to ask wheter "start from neck-created URL again(from scratch)" is mandatory in this bug's case or not.

(Q4) "where am I?" is already known in this bug's case, because request while processing an HTML mail in currently opened mail folder. If caller of "read of whole mail data in mail folder file" passes mURI to handler of "open of relevant mail folder/mail folder file", I think "matching with all mail folder files" is not needed.
Is it right?
(In reply to comment #90)
> (snip) because a mailbox URL looks like
> "mailbox:///C|/.../Thunderbird/Profiles/bejnf55f.Test1/Mail/Local%20Folders/Trash?number=0&part=1.3&filename=Teil%201.3.gif"
> -- i.e. it doesn't contain information about which folder we're in, only the
> path on the hard drive.

(Q5) Can process like next be used for conversion from "neck-mailbox: URL" to mURI? (internal file path -> internal folder path)

> mailbox:///C|/... /Mail/Local%20Folders/P1.sbd/P2.sbd/P3.sbd/Folder?number=12345&part=1.3&filename=Teil%201.3.gif
> => /C|/... /Mail/Local%20Folders/P1.sbd/P2.sbd/P3.sbd/Folder.msf => folder name of "Folder"
> => /C|/... /Mail/Local%20Folders/P1.sbd/P2.sbd/P3.msf => folder name of "P3"
> => /C|/... /Mail/Local%20Folders/P1.sbd/P2.msf => folder name of "P2"
> => /C|/... /Mail/Local%20Folders/P1.msf => folder name of "P1"

If folder-name != file-name, file-name.msf exists, and folder-name is held in folder-name.msf. If folder-name == file-name and file-name.msf exists, no problem. If file-name.msf doesn't exists folder-name == file-name.

By the way, no performance problem was observed with HTML mail in next folder "A", with local profile/local mail folder.
  A    : HTML mail with img => no problem
  AAAA : subfolder of 0000 to 9999 (...\AAAA.sbd\0000 to ...\AAAA.sbd\9999)
  ZZZZ : same HTML mail with img as one in A => problem is observed.
=> Holding all HTML mail with img in folder "A" is a workaround :-)
(In reply to comment #94)
> (In reply to comment #93)
> > I call these URLs "necko-created". Don't get me wrong here -- I'm not saying that necko is in any way to blame.
> 
> Oh, sorry for my bad question. I wanted to ask wheter "start from neck-created
> URL again(from scratch)" is mandatory in this bug's case or not.

I'm 99% sure it is, yes. I believe you need to create new URLs for every image you load.

> 
> (Q4) "where am I?" is already known in this bug's case, because request while
> processing an HTML mail in currently opened mail folder. If caller of "read of
> whole mail data in mail folder file" passes mURI to handler of "open of
> relevant mail folder/mail folder file", I think "matching with all mail folder
> files" is not needed.
> Is it right?

Yes, but only URLs created by mailnews know how to do that. The URLs for images are created by the docshell.

(In reply to comment #95)
> (In reply to comment #90)
> > (snip) because a mailbox URL looks like
> > "mailbox:///C|/.../Thunderbird/Profiles/bejnf55f.Test1/Mail/Local%20Folders/Trash?number=0&part=1.3&filename=Teil%201.3.gif"
> > -- i.e. it doesn't contain information about which folder we're in, only the
> > path on the hard drive.
> 
> (Q5) Can process like next be used for conversion from "neck-mailbox: URL" to
> mURI? (internal file path -> internal folder path)
> 

No, not really, there are a bunch of problems with that approach (including bug 497680). The only safe and known correct approach seems to be to match entire folder paths (David, please correct me if I'm wrong here).
can someone clearly state the workaround for this issue and point to it via whiteboard? thanks.

also, perhaps someone can catch up with bug 542747
again ...
> can someone clearly state the workaround for this issue and point to it via
> whiteboard? thanks.


sid0 in comment #71
> I can believe that my patch caused the regression here -- using a debugger +
> procmon, I found that each equals comparison causes a call to GetShortPathNameW
> at
> <http://mxr.mozilla.org/comm-central/source/mozilla/xpcom/io/nsLocalFileWin.cpp#3053>,
> which in turn opens the file and whatever else is mentioned in comment 60.
> Since this is essentially O(N) random disk I/O in the total number of folders
> you have, it could potentially hurt a lot.
bug 497680 set as regressor, checkin date 2009-06-19. please correct if not accurate

> Since for correctness purposes it seems like we need to compare every folder,
> what we seem to need here is a map from folder paths to folders, probably
> implemented as a hash table. nsIFolderLookupService maybe? David, any other
> ideas to reduce the number of folders to compare here?

bienvenu in comment #75
> sid0, this would be a good reason to revive the folder lookup service...

bienvenu, sid0...what are our options to resolve this in the 3.1 time frame?  is folder lookup service a long shot?  what are the implications of backing out bug 497680? (it's not clear to me what user facing problems this fixed)


renominating blocking because issue can be severe, eg minutes, (hopefully will have this better quantified by end of week if the people in http://gsfn.us/t/o85n are cooperative), a regression, and seems on par with if not worse than some bugs on the 3.1 blocking list, like bug 549931   


do replies (compose) to affected message have the same issue?  there might be a couple duplicate bugs, bug 542747 for example. But I'm not finding lots of examples from the list of performance bugs [1].

[1] https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=anywordssubstr&field0-0-0=short_desc&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&type1-0-1=substring&resolution=---&value1-0-1=perf&classification=Client%20Software&classification=Components&chfieldto=Now&chfield=[Bug%20creation]&query_format=advanced&chfieldfrom=2009-06-19&value1-0-0=slow&type0-0-0=nowords&value0-0-0=count%20counts&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=keywords
Blocks: 497680
blocking-thunderbird3.1: --- → ?
Component: Message Reader UI → Backend
Product: Thunderbird → MailNews Core
QA Contact: message-reader → backend
Summary: HTML Mail View very slow (45 sec - 1min) → HTML Mail View very slow (45++ sec) for pop and local folders
Version: 3.0 → 1.9.1 Branch
As suggested here is some info about my case:

1. state in what folder types the issue is seen - pop, imap or local folder accounts
CK: Occurs in local folders (I do not use pop, imap or other folder types for direct operations)

2. is message compose affected. in other words, for an affected message after it is selected and fully displayed, is the time to bring up the reply window or to send the reply also extremely slow
CK: Yes, starting a forward or a reply also takes a long time for long emails. Looks like it's the same delay here.

3. for the typical message you view that is affected, what is your *average* response time to display a message
CK: 5 to 10 seconds (for affected emails, other emails do not show any delay; also no delay when I remove large top level folders (several GB).)

4. what is the *worst case* message display time you have encountered
CK: Guessing: 30 seconds
One simple fix might be to cache the most recent request + answer since I suspect in the bad perf case, we ask the same question over and over again. It's a little ugly since we'd need to use a static var.
Attached patch wip patch — — Splinter Review 
I won't be able to get to this before July -- would you like to take over the bug, David?
This is a nasty regression for anyone that uses networked file stores on Windows, I think it needs to block.  Giving to David in the hopes that he's up for taking it over...
Assignee: nobody → bienvenu
blocking-thunderbird3.1: ? → rc1+
over IRC, Sid and I agreed that caching the most recent result would be the simplest ameliorating fix for 3.1, and we should get Sid's fix for 3.2 - he's worried that his fix won't work on 1.9.2 on the mac, in some situations, anyway.
What sucks a bit is that I can't use a static nsCString, so I need to hang the nsCString off of some service. The account manager is the most reasonable place to do this, until Sid's patch, when we can hang it off the folder lookup service, or not need to hang it anywhere...but that's going to lead to a little churn in the account manager interface. Probably can't be helped...I looked at using a module destructor to free the memory, but base/utils isn't a module. We're having a similar issue with trying to do size strings in a shareable way.
This moves the function into the account manager, and caches the most recent query and result.

I'm not sure how much sense a unit test for this would be, since it wouldn't tell you if the result was cached or not, but I guess I'll write one that exercises the code path, at least, if there's not already a test that does so.

I'll also request a try server build for anyone having this issue to try.
try server builds should be up in a couple hours, if all goes well - http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry
reporters,
please test and report your experience as to whether the following builds work or not. These are builds based on version 3.1beta which have NOT gone through QA testing - backup your profile before using. Note if you are using v3.0 with indexing turned on, your (gloda) search index will be rebuilt. This is normal. Also, do not expect all v3.0 add-ons to work in v3.1

Mac http://s3.mozillamessaging.com/build/try-server/2010-04-28_17:10-bienvenu@nventure.com-1272499552/bienvenu@nventure.com-1272499552-mail-try-mac.dmg
windows http://s3.mozillamessaging.com/build/try-server/2010-04-28_17:10-bienvenu@nventure.com-1272499552/bienvenu@nventure.com-1272499552-mail-try-win32.installer.exe

I posted the tryserver links in gsfn
Attachment #442253 - Flags: superreview?(bugzilla)
Attachment #442253 - Flags: review?(bugzilla)
Comment on attachment 442253 [details] [diff] [review]
[checked in] proposed fix for 3.1

Ok, this looks good. r/sr/a=Standard8 (I'll land in a bit).
Attachment #442253 - Flags: superreview?(bugzilla)
Attachment #442253 - Flags: superreview+
Attachment #442253 - Flags: review?(bugzilla)
Attachment #442253 - Flags: review+
Attachment #442253 - Flags: approval-thunderbird3.1+
Marking as fixed for beta 2. I don't know if David wants to do the folder lookup service follow-up in this bug or a different one...
status-thunderbird3.1: --- → beta2-fixed
(In reply to comment #111)
> I don't know if David wants to do the folder lookup service follow-up in this bug or a different one...

Even if this bug will be resolved by caching of file handle, performance issue produced by change of bug 497680 really exists in next sitution(very many mail folders are used by user).
  Cretate a mail folder A, create mail folders of 0000 to 9999 in folder A.
  => Restart of Thunderbird takes very long, and CPU utilization while 
     restarting Tb becomes very high, even though only local files is used,
     especialy in MS Win environment due to not-so-efficient directory
     access on MS Win.
I think this kind of issue is better to be processed in different bug.
just tested this with lanikai nightly from April 30:
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.5pre) Gecko/20100430 Lanikai/3.1pre

and all the test case .emls i.e. case0, case1 and case 2 open immediately with no delay
I'd be surprised if the fix was in the nightly build.

Re WADA's comment, I don't think that code runs on startup, so I'm not sure what issue he's seeing.
(In reply to comment #114)
> Re WADA's comment, I don't think that code runs on startup,
> so I'm not sure what issue he's seeing.

Who saw very long startup time if 10,000 null local mail folder files are used by Tb, was me. I checked "10,000 null local mail folders before local mail folder which has HTML mail with some <img>s" case for this bug on MS Win-XP.
I'll open separate bug for very long startup time issue with 10,000 local mail folders, after some additional checkings.
It may be simply a performance issue of MS Win when acquiring many file names in a directory, because I saw slowness of SysFileTree("<dirname>",files,"S"); of REXX on MS Win(similar to DIR /S. puts file names/file's meta data such as size/timestamp in array of "files") when many files exist in a directory(it was roughly 10,000 files).
So, yes, we have to iterate over the contents of the local folders during startup to discover what the users' local folders are. That hasn't changed in a very long time, and I doubt its performance has changed between TB 2 and TB 3 (though if there's evidence to the contrary, I'd be interested.)
(In reply to comment #114)
> I'd be surprised if the fix was in the nightly build.
> 
oops yes, you are right, i doubt the fix is in the nightly build

i also cannot reproduce this problem on windows 7 TB 3.0.4 (1 gmail pop, 1 gmail IMAP, 1 hotmail account) with the attached test emails but this is probably an irrelevant datapoint since WADA and others can reproduce it (see comment #82)
I believe the key to reproducing this is to have lots (thousands) of folders.
(In reply to comment #117)
> i also cannot reproduce this problem on windows 7 TB 3.0.4

Roland Tanglao, please read bug 536873 comment #30. If you create 10,000 null local mail folders before local mail folder in which HTML mail of several to many <img src="cid:..."> tags, you can easily observe performance issue of this bug(you can take short coffee break while Tb is trying to display images), with local file only, without remote file/Offline Folder(CSC of Win).
My quick unscientific analysis:
Mail with 12 embedded images, 7 folders deep.
3 seconds+ with:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100324 Shredder/3.0.5pre ID:20100324001346
About 1 second with:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.5pre) Gecko/20100501 Lanikai/3.1pre ID:20100501031733

66% perf gain

Thanks David
Checked with next build.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.3a5pre) Gecko/20100501 Shredder/3.2a1pre

Local profile, local mail folders only.
  Folder AA : Test HTML mails attached to this bug
  Folder AB : subfolders from 0000 to 9999 (10,000 subfolders)
  Folder ZZ : Test HTML mails attached to this bug (same content as AA)
Test result.
- At both AA and ZZ, HTML mails are displayed at a glance.
- No difference was observed between AA and ZZ, except very short time lag
  upon first open of folder ZZ followed by first HTML display.
=> VERIFIED
probably want to clone this for the 3.2 version of the fix.
Whiteboard: [gs] → [gs][ameliorating fix landed for 3.1 b2]
Blocks: 563677
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
(In reply to comment #122)
> probably want to clone this for the 3.2 version of the fix.

Is bug 563677 for the 3.2/trunk patch? 

And regarding 3.0.x, I though this was fixed in .5, but per IRC with bienvienu, we missed getting a fix into 3.0.5 for this bad UE. Perhaps fell off the radar because we were thinking 3.1 would ship first. Plus there's no bug marked blocking 3.0 for this behavior. Are we going to fix 3.0 off of this bug or spin a new bug?
blocking? for 3.0(.6) so we don't lose track (if this turns out to be the right bug for the patch)
blocking-thunderbird3.0: --- → ?
(In reply to comment #124)
> blocking? for 3.0(.6) so we don't lose track (if this turns out to be the right
> bug for the patch)

Unfortunately the required changes are not something we can take on the 3.0.x branch, so we can't fix this bug there. Thunderbird 3.1 should be out very soon, so it is suggested that users seeing this upgrade to that.
blocking-thunderbird3.0: ? → -
status-thunderbird3.0: --- → wontfix
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: