589798 - Local folders on a Windows net drive offers an extremely slower access than TB15 & TB2, slow access to messages, folder, attachment

Status: RESOLVED WORKSFORME

(Thunderbird :: General, defect)

Description

[Laurent Bauvens]

User-Agent:       Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b3) Gecko/20100805 Firefox/4.0b3
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.8) Gecko/20100802 Thunderbird/3.1.2

A lot of our users have relocated their Local folders account on Windows servers in order to have a administrative backup of their old messages.

With Thunderbird 1.5 and 2.0, they were happy users but with TB3, they enter hell because the access to their messages is so slow that Thunderbird sometimes fails to display a folder content or even an attachment.

GLODA is off on our TB configurations. My test mailbox is huge (1,6 Gb) and contains lot of folders. But some users experience the issue with a size of mailboxes of only 100 Mb.

On the serverside (my test "server" is a PC Intel Celeron 325@2.5Ghz+512Mb of RAM + WinXP Pro SP3 with a shared folder and the client PC is a WinXP Pro SP3+2Gb of RAM), I personally view that a single request on an attachment can make up to 50% of CPU charge during few seconds. A quasi DoS attack ! The same action with a previous version (<3.0) is really imperceptible in the task manager CPU display of the server.

With Process Explorer (a MS Sysinternals utility) on the client, I saw that TB3 opens only few index files and mail databases whereas older TB versions open all index files.

Other issue (related or not), when I click on another folder than the current one, Thunderbird sometimes seems to do nothing (or wait something) for a long time (~20+ seconds) before to display the content of the folder. In this case, nothing is relevant on Process Explorer.

Reproducible: Always

Steps to Reproduce:
1.Share a folder on a Windows Server (\\server\mylocalfolders)
2. On the Windows client, connect a net drive to the shared folder (X: -> \\Server\mylocalfolders
3.With the file manager, copy the files of a complex Local Folders account on the shared folder
4.Start TB and modify the location of the Local folders account. Deactivate GLODA indexing engine.
5.Restart TB
6.Browse folders and messages, open attachments.

Actual Results:  
Access to certains folders is slow, very slow and, in extreme cases, fails. Server is overloaded

Expected Results:  
Access to all folders, messages and attachments is quick without overloading the server.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.8)
> Gecko/20100802 Thunderbird/3.1.2

> A lot of our users have relocated their Local folders account on Windows
> servers in order to have a administrative backup of their old messages.
> With Thunderbird 1.5 and 2.0, they were happy users but with TB3,
> they enter hell because the access to their messages is so slow
> that Thunderbird sometimes fails to display a folder content or even an attachment.

Tb 3.0's slowness(Tb 3.0.4 or before) in file I/O(write) of mail folder file due to "write request per a mail data line" is already known and is already fixed problem. See bug 539389, which is fixed by Tb 3.0.5/Tb 3.1.0 or later. Problem of bug 539389 is easily be seen by Process Monitor log.

Does problem what you reported to this bug still occur with Tb 3.0.5/Tb 3.1.0 or later?
If so, what is difference of Process Monitor log between "Tb 3.0.0-3.0.4" and "Tb 3.0.5 or later/Tb 3.1.0 or later"?
(What is difference of this bug from bug 539389?)

Comment 2

[Laurent Bauvens]

(In reply to comment #1)
> Does problem what you reported to this bug still occur with Tb 3.0.5/Tb 3.1.0
> or later?

Yes. I use a Thunderbird 3.1.2 for my tests.

> If so, what is difference of Process Monitor log between "Tb 3.0.0-3.0.4" and
> "Tb 3.0.5 or later/Tb 3.1.0 or later"?
> (What is difference of this bug from bug 539389?)

I'm going to study and compare bug 539389 to answer ASAP.

Comment 3

[Laurent Bauvens]

OK I read the bug 539389 and perform some additional tests with Thunderbird 1.5, 3.0.4 and 3.1.2.

I'm very surprised by the results.

Double-clicking on an attachment of about 2156 kb (MIME encoded, real 1621 kb)
3.0.4 .... 37 s 
3.1.2 .... 10 s
1.5.0.2 .. less than 1 s !

On serverside, the CPU usage is generally long (seconds) and high (60 .. 100%) for versions 3.x.x ; and short (1 sec or less) and high (40 .. 100%) for 1.5 series. 

So the fix for bug 539389 has definitely permit to retrieve a user experience close to normal but yet far from 1.5 series performance. 
Concerning the high server cpu usage, it will deserve to be drastically reduced. I will try to have some results from a real Windows server (I test on a pseudo server under WinXP Pro).

Concerning the issue related to the folder browsing, I think it comes from non indexed folders in my test mailbox, once indexed the folder browsing experience is better.

Comment 4

[Laurent Bauvens]

Attachment: Screenshots of Process Monitor to show different manner to make a download between TB15 and TB312

Comment 5

[Laurent Bauvens]

I perform others additional tests with Sysinternals Process Monitor.

So I note something strange for me: 
 - TB15 download 11x4096 bytes in a sequential manner and flush data in tmp file by 4x8192 bytes + 1x4 bytes
 - TB3 download 11x4096 bytes in a multi-offset sequential manner and flush data in tmp file by 1x32768 bytes

I think multi-offset data download could produce high CPU usage on the server.

-> See attachment 471122 [details]

Comment 6

[Ludovic Hirlimann [:Usul]]

Laurent do you think you could produce  a patch to fix this ?

Comment 7

[David :Bienvenu]

Laurent, can you describe specific things that are slow now, since your initial report seems to describe 3.0?

Comment 8

[Laurent Bauvens]

Attachment: CPU graph of a Windows Server which well describes the problem

This Windows Server hosts mainly Local folders of certain users who use principally Thunderbird 1.5 and 2.0 series. Only one uses TB in version 3.0.6.
The high cpu usage painted in orange represents this user double-clicking on a simple attachment. Size of attachment isn't enormous as our messaging server limits the message size at 4 mb.
This graph confirms the same behavior I see on my test server.

Comment 9

[Laurent Bauvens]

(In reply to comment #6)
> Laurent do you think you could produce  a patch to fix this ?

Sorry, I couldn't produce a patch because I'm not a developer (just an IT crew member in charge of studying Thunderbird) and my organization doesn't have any development resource with knowledge in Mozilla coding. :(

Comment 10

[Laurent Bauvens]

Hi David

(In reply to comment #7)
> Laurent, can you describe specific things that are slow now, since your initial
> report seems to describe 3.0?

My initial report refers to a version 3.1.2 not a 3.0. But my TB3 test users work with a 3.0.* (currently a version 3.0.6).

After fixing bug 539389 in 3.0.5, the point of view of these users hadn't changed: "Access to Local folders is always very slow, a real nightmare" (workaround: they copy message into a folder of their mailbox to access quickly to an attachment)

To this slowness feeling from my test users, I add the potential serverside issue I found.

I have to say that bug is a real problem for us because we had planned to start deploying Thunderbird 3.*.* early in september. In fact the deployment is now postponed.

Comment 11

[David :Bienvenu]

I can look into the opening of attachments issue...

Comment 12

[Laurent Bauvens]

(In reply to comment #11)
> I can look into the opening of attachments issue...

Yes, it would be very appreciated that you spend some time to have a look on this issue.

Thanks. :)

Comment 13

[Laurent Bauvens]

Concerning the folder browsing issue I mentioned in comment #3, I now note that (with TB 3.1.3, but perhaps also with previous versions) when I click on  folders, TB seems frozen (UI is totally unclikable) for seconds. On server side, I saw the index file related to the folder has a size of 0 byte. When TB returns to a normal usability state, the list of messages appears and index file size is > 0 byte. 

But when I open the error console, it contains this:

Error : uncaught exception: [Exception... "Component returned failure code: 0x80550006 [nsIMsgFolder.msgDatabase]"  nsresult: "0x80550006 (<unknown>)"  location: "JS frame :: chrome://messenger/content/mailWidgets.xml :: parseFolder :: line 2141"  data: no]

Does it deserve to file a new bug or not ?

Thanks.

Comment 14

[David :Bienvenu]

Laurent, could you do the measurements in https://bugzilla.mozilla.org/show_bug.cgi?id=589798#c3 for 3.0.5? 

Is this bug open essentially because 3.1 takes 10 seconds vs 1.5 taking less than 1 second? Or does 3.0.5 take 37 seconds? 3.0.5 doesn't have all the fixes that 3.1.x does.

Re comment 13, somehow your .msf file is either getting corrupted (e.g., timestamp is changing dramatically on the corresponding folder, by more than an hour, or the file size is wrong). Either will cause us to rebuld the .msf file. Do you have any particular steps that cause us to rebuild the summary file?

Comment 15

[David :Bienvenu]

Laurent, what kind of attachment are you opening? This is from a local folder, not an imap folder configured for offline use, right? I see 32Kb reads from the local mail folder and 32kb writes to the tmp file, when opening a .png attachment from a local mailbox. I also don't know what you mean by multi-offset - I'm not seeing the systinternals process monitor say anything about multi-offset writes.

Comment 16

[David :Bienvenu]

3.1.3 does seem to do 4K reads, but trunk builds are doing 32K reads. We're telling the pipe code to use 8 4k segments in both trunk and 3.1 code. I'm not sure why that would result in different read counts.

Comment 17

[Wayne Mery (:wsmwk)]

Laurent, in what areas do you seen improvement when using v3.1.5 now that Bug 599119 is fixed?  and what areas are still bad?   (and comment 15)

Comment 18

[Laurent Bauvens]

First of all, I must apology to not feed this bug since a month.

So, since the version 3.1.3, the issue concerning a slow access to attachment was dramatically reduced. It's definitely not as speedy as in 1.5 or 2 series but ...

A/ attachment viewing is now useful (at least in users' feeling) because attachment is now always opened, and in an acceptable time lapse; and 

B/ the attachment's opening now requires less CPU usage & time on the Windows server.

Nonetheless, in the current version (3.1.5), it remains the issue concerning the self re-indexing process of a folder which always freezes TB during dozens of seconds without any relation with the size of the message database file. For example, I experienced the case a 37 kb file re-indexing which has needed almost 40 seconds and during this time, TB seemed frozen for the user.

Concerning 3.0.x series, nothing has been fixed related to this present bug.

Comment 19

[Wayne Mery (:wsmwk)]

(In reply to comment #18)
> First of all, I must apology to not feed this bug since a month.
> 
> So, since the version 3.1.3, the issue concerning a slow access to attachment
> was dramatically reduced. It's definitely not as speedy as in 1.5 or 2 series
> but ...
> 
> A/ attachment viewing is now useful (at least in users' feeling) because
> attachment is now always opened, and in an acceptable time lapse; and 
> 
> B/ the attachment's opening now requires less CPU usage & time on the Windows
> server.

thanks for the feedback. adjusting severity to normal.

do you see your attachment issue described in any of the bugs in this list? ..
https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=substring&keywords=perf&keywords_type=allwords&field0-0-0=short_desc&bug_severity=blocker&bug_severity=critical&bug_severity=major&bug_severity=normal&bug_severity=minor&type0-0-1=substring&field0-0-1=component&type1-0-1=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&query_format=advanced&value0-0-1=attach&type0-0-0=anywordssubstr&value0-0-0=attach&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=short_desc

also, please reply about comment 15


> Nonetheless, in the current version (3.1.5), it remains the issue concerning
> the self re-indexing process of a folder which always freezes TB during dozens
> of seconds without any relation with the size of the message database file. 

I suspect this is covered by other bug(s). bienvenu may be able to suggest one

> Concerning 3.0.x series, nothing has been fixed related to this present bug.
I wouldn't expect v3.0 to be changed. Main focus is currently on v3.1 and its successor

Comment 20

[L A Walsh]

Please consider making the network and disk I/O sizes adjustable in the 'registry' (about:config settings)...

Ideal size for my local net is about 16MB read and write.  32K is VERY slow -- 4K is even slower.

The problem described happens when you talk to a local IMAP server (which also has small I/O sizes).  

Given Tbird3.0's default of downloading everyone's IMAP folders from their local network server to their 'home dir' (also on the local network server for the person in this bug), I can't imagine Tbird being usable.


In my case, My local home dir is in my 'roaming dir', which meant that tBird3.0 downloaded almost 2GB of indexed IMAP folders onto my local home dir, which then added 30 minutes to my logout time (and about 3minutes to login).  

Took a while to catch the reason, but reverting to 2.0 solved the problem.

Comment 21

[Wayne Mery (:wsmwk)]

LA Walsh, Laurent, this is still a problem in *current* versions, eg version 8 and newer, correct?

Comment 22

[Wayne Mery (:wsmwk)]

is the .msf rebuild issue possibly Bug 232047??  Frequent need to rebuild summary file(.msf) when mail folder is located on a different OS's sharing hard disk or network drive(timestamp inconsistency)


(In reply to L A Walsh from comment #20)
> Please consider making the network and disk I/O sizes adjustable in the
> 'registry' (about:config settings)...

this is wada's Bug 558528 - Consider larger buffer size in file I/O by Mail&News. Use of Necko's buffer size, buffer size user can easily customize, .

=> severity=major based on comment 3 measurements (and assuming they are still the same)

Comment 23

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #21)
> LA Walsh, Laurent, this is still a problem in *current* versions, eg version
> 8 and newer, correct?

Comment 24

[Laurent Bauvens]

(In reply to Wayne Mery (:wsmwk) from comment #21)
> LA Walsh, Laurent, this is still a problem in *current* versions, eg version
> 8 and newer, correct?

Concerning the issue with message bases on Windows shared folders, Thunderbird 17 doesn't experience any slow down.

Concerning the re-indexing issue which froze Thunderbird 3.*, I currently haven't time to make an in depth test but our internal beta-testers don't experience this kind of freeze with the lastest Thunderbird version (17).

Comment 25

[Wayne Mery (:wsmwk)]

Laurent, thanks for the update.
WFM per reporter