563677 - HTML Mail View very slow (45++ sec) for pop and local folders - speed up FolderUriForPath first lookup when switching folders

Status: NEW

(MailNews Core :: Backend, defect)

Description

+++ This bug was initially created as a clone of  +++

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

Some HTML Mail freezes Thunderbird for a while.
Mostly in large email folders but not necessarily a large email.
Emails came from Lotus-Notes-Client and contains only few text and some spacer GIF for layout.

Effect also occurs in '-safe-mode' of Thunderbird.
No anti-virus programm is activ.
Yes, Profile is in network folder but this is probably not the reason.
Text-View of same email is OK.
Only HTML view appears as problem.

CPU-use graph in Windows-Task-Manager shows very high peaks for a while.


Reproducible: Always

Steps to Reproduce:
1. create a larger Emailfolder (about 200 plus xx)
2. ask to get HTML-mails from different Lotus Notes User
3. Open Task-Manager for CPU-use
5. switch preview of emails and notice what happens

Actual Results:  
Email is not shown for a while. 
Can't change to another mail or function.

Expected Results:  
show the email content in preview window.

Comment 1

[WADA:World Anti-bad-Duping Agency]

David, same problem as Bug 545126, isn't it?

(Time to take display an HTML mail) == A * B * C,
 where;
  A : Number of <img src="cid:..."> in an HTML mail
  B : Number of local mail folders before folder in which the HTML mail is kept
  C : Time to acquire directory information of a mail folder file

(Case-1) Original problem on bug opener of Bug 545126
  A: Large(Lotus Notes mail), B: Large, C: Large(remote profile, CSC of Win)
(Case-2) Minimized test case by bug opener of Bug 545126
  A: Small(reduced to 7), B: Large(same as Case-1), C: Large(same as Case-1)
(Case-3) My test with 10,000 local mail folders with test mail I attached
  A: Small(4 or 40), B: Very Large(10,000), C: Small(local profile, no CSC)
(Case-4) Bug 536873 (DUP'ed to Bug 545126)
  A: Medium, B: Larger than Case-1/Case-2, C: Small(local profile, no CSC)
(Case-5) This bug
  A: Large or Very Large (Lotus Notes mail),
  B: Medium to Large (200?),
  C: Not Small (remote profile, CSC?)

Is cached result(by change to "caching the most recent result", Bug 545126) lost if very many <img> exist(Case-5)?
Caching doesn't work well if special mail structure(not-well-formed structure, funny nesting of multipart)?
Or caching doesn't work well if wrong Content-Type?
(application/octet-stream for image data part with Content-ID:)

Comment 2

[David :Bienvenu]

WADA, I cloned bug 545126 so that we could do a non-caching fix, using sid's patch as a starting point. Caching doesn't help the first lookup, but subsequent lookups should be fast. This bug is about fixing the first lookup.

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to comment #2)
> WADA, I cloned bug 545126 so that we could do a non-caching fix, using sid's
> patch as a starting point. Caching doesn't help the first lookup, but
> subsequent lookups should be fast. This bug is about fixing the first lookup.

bienvenu, standard8, is this bug still wanted, since the 3.0->3.1 major update has flown the coop?

Comment 4

[David :Bienvenu]

yes, if I remember, sid had some ideas about how to make the first lookup faster using a hash table that we keep up to date.

Comment 5

[Wayne Mery (:wsmwk)]

If someone sees slow message display on first view, but second and subsequent views are fast, then then there's a good chance they are seeing this bug?


sid, possible for you to revisit this soon?

Comment 6

[Joe Sabash [:JoeS1]]

I have a similar problem which may be related to Wada's Point B above.
B : Number of local mail folders before folder in which the HTML mail is kept
It shows itself with simple folder selection (a newsgroup about 150 folders deep)
Can take up to 30 seconds to select and show the threadpane.
Heavy disc I/O during that delay.
New bug..or related.

Comment 7

[David :Bienvenu]

(In reply to comment #5)
> If someone sees slow message display on first view, but second and subsequent
> views are fast, then then there's a good chance they are seeing this bug?
Yes, on a per-folder basis, iirc.  The first lookup from the first message loaded in a folder is slow; subsequent lookups on that message (i.e., the message triggers multiple path lookups), and lookups from other messages in that same folder should be fast, until you click on an other message in an other folder that does an other lookup, changing the cached lookup.

Comment 8

[Wayne Mery (:wsmwk)]

ah, as described by http://getsatisfaction.com/mozilla_messaging/topics/slower_performace#reply_4890042 for example

that's pretty severe. 

No workaround? Like keeping each affected folder open in a tab?

Comment 9

[Joe Sabash [:JoeS1]]

(In reply to comment #8)
> ah, as described by
> http://getsatisfaction.com/mozilla_messaging/topics/slower_performace#reply_4890042
> for example
> 
> that's pretty severe. 
> 
> No workaround? Like keeping each affected folder open in a tab?

Yes, keeping the affected folder open in a tab fixes the problem for me.
However, it adds a problem. The msf files are loaded into memory. For system with limited memory (large folders) this is can be problematic.

Example:
1 tab open TB=86,280
4 tabs     TB=302,800

On a 512 meg box, you won't be doing much else with this workaround.

Comment 10

[Wayne Mery (:wsmwk)]

this should block - or at least get focus via "needs" - there are people leaving version 3 for version 2, etc because of this bug.

Comment 11

[Mark Banner (:standard8)]

I don't think we can block 3.3 on this, mainly as we haven't got anyone to work on it and we don't have long before we branch - though it is definitely wanted (also, whilst I realise some users are affected, I'm not sure that it is significant enough that this is a stop-ship).

Comment 12

[Wayne Mery (:wsmwk)]

there are a few more reports of this on gsfn

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to David :Bienvenu from comment #4)
> yes, if I remember, sid had some ideas about how to make the first lookup
> faster using a hash table that we keep up to date.

Is it not time to dust this off?

I don't have time ATM to benchmark the current state of slowness, but I have to think that users are still seeing this.

Comment 14

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to Joe Sabash [:JoeS1] from comment #6)
> I have a similar problem which may be related to Wada's Point B above.
> B : Number of local mail folders before folder in which the HTML mail is kept
> It shows itself with simple folder selection (a newsgroup about 150 folders
> deep)
> Can take up to 30 seconds to select and show the threadpane.
> Heavy disc I/O during that delay.
> New bug..or related.

Joe, can you profile using  https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Thunderbird ?

possible report http://forums.mozillazine.org/viewtopic.php?f=39&t=2345081&p=11618255#p11618255

Comment 16

[Joe Sabash [:JoeS1]]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #15)
> (In reply to Joe Sabash [:JoeS1] from comment #6)
> > I have a similar problem which may be related to Wada's Point B above.
> > B : Number of local mail folders before folder in which the HTML mail is kept
> > It shows itself with simple folder selection (a newsgroup about 150 folders
> > deep)
> > Can take up to 30 seconds to select and show the threadpane.
> > Heavy disc I/O during that delay.
> > New bug..or related.
> 
> Joe, can you profile using 
> https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Thunderbird ?
> 
> possible report
> http://forums.mozillazine.org/viewtopic.
> php?f=39&t=2345081&p=11618255#p11618255

Well, I am no longer reproducing the effects of this bug on any of my PC's with Trunk or any currently tested versions. Folder selection is less that a second for me.

Comment 17

[:aceman]

The folder lookup also changed in some aspects (like the RDF removal) since this bug was filed.
If we aren't getting any more more reports of this in the last years I suggest we could close it WFM.

Comment 18

[Wayne Mery (:wsmwk)]

Mike, are you still able to reproduce this?

Comment 19

[Wayne Mery (:wsmwk)]

Still occurs, at least the testcase in bug 1512974.

It would be good for someone to get a profile using https://github.com/thunderbird-conversations/thunderbird-conversations/wiki/Profiling-Conversation's-Performance against a reduced testcase would be good to have.