Closed Bug 815730 Opened 12 years ago Closed 10 years ago

TB16 Redownloads messages / Constantly bringing folders "up to date" / horrible IMAP performance

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 816327

People

(Reporter: rkent, Unassigned)

References

Details

(Keywords: perf, Whiteboard: [has protocol log:comment 11])

Attachments

(2 files)

Thunderbird 17 log (IMAP attached file corrupt)
1.29 MB, application/octet-stream
Details
Thunderbird 17 IMAP logs (IMAP attached file corrupt)
38.89 KB, application/octet-stream
Details 
+++ This bug was initially created as a clone of Bug #806760 +++

Since bug 806760 got narrowed to the Zarafa issue, but others are still seeing issues with the same description, I am cloning that bug to discuss any additional issues. The status of this bug is starting at UNCONFIRMED until we have some reproducible test cases that we can work on. If there is an existing bug please point me to it instead.

As with the previous bug, any help in giving a developer access to an account that can reproduce this problem will go a long way toward getting this resolved.

Ideally, if you are able to provide a reproducible case, please try to setup a test account that has any required messages or folders that cause the bug already loaded, and will reproduce the problem when the account is installed from a new profile. I realize that is a pretty high bar, but from my experience with trying to reproduce the previous bug is that it is not easy to generate similar conditions from verbal descriptions in comments.

Any help in this is greatly appreciated. If you cannot provide the test report, please at least provide a new IMAP log using TB 17 for a case where the error is occurring.

Florian, it would be particularly valuable if you could setup a test account that demonstrates the problem using a fresh profile, as you both continue to see issues and have been willing in the past to provide a test account. If you do so, document any changes in setup that are needed from default to cause the error to occur.
I use Thunderbird 17 on Japanese Windows 7 (64bit) with IMAP.
Sometimes,  I received mails with corrupted attached files.
(not every time with same file.
 When I run "repair folder" command, I can get correct attached file.)

I got Thunderbird log when I received a test mail from myself with a corrupted attached file.
I hope this log helps to debug...
(I deleted some sensitive data from this log.)

(Our IMAP server is "DEEPSoft MailSuite".
 http://www.deepsoft.co.jp/ )
(In reply to Makoto Itani from comment #1)

I think it is quite important to keep an eye on this.
When we installed Seamonkey 2.13.1 (comparable to Thunderbird 16.0.1) in the company, we
got several reports of this issue.  We use UW IMAPD so likely it is not immediately related
to the IMAP server in use.
I had one message where it could be sort of reproduced, and now that I have installed
Seamonkey 2.14 (comparable to Thunderbird 17) on my own workstation I cannot reproduce
the issue with that same message.  I have now asked another expert user to test with 2.14
to see if issues still occur.

Do you have a local copy of the mail in your profile (offline cache)?
We have disabled this caching as much as possible as we do not like to have the copy of
all mail in the user's profile.  Mail should always remain on the (company) IMAP server
and be fetched (over the fast LAN) for every operation.  So in the ImapMail directory we
have only .msf files.   The setting (caching on or off) also affects the circumstances
that appear to cause this issue.
Keywords: perf
Quick update from my side:

It seems what I'm seeing is a double-effect.
On one hand, TB 17 seems to get sluggish after a while due to one of my addons. I'm currently trying to find out the culprit by selectively enabling and disabling extensions, but have no good guess yet.

However, the redownload phenomenon seems to be there as well. It happens rarely, not at all like with TB 16, where it was permanent. I tried to produce a log, but - Murphy's everywhere - everytime I had the log running, of course the issue did not appear.

I will try to hunt down both issues and update this issue asap.
Using trunk, lately** i'm seeing occasional "bringing up to date" of All Mail and sometimes it claims it is redownloading 100+ messages but it's not the entire folder. A couple of times that DL has "hung" at the first message for some long period of time (upwards of a minute or two) with the activity mgr progress bar spinning and the status bar spinning - which sucks CPU..

** after I changed gmail account's drafts to point to [Gmail]/Drafts
(In reply to Rob Janssen from comment #2)

Thank you very much.

I tried following two settings on Thunderbird 17.

+ uncheck the checkbox "keep messages for this account on this computer"
+ check the checkbox "Don't download messages larger than 50KB"

Both of settings, I received mails with corrupted attached files...
But when I restart Thunderbird once or twice, I can get correct attached file without "repair folder" command.
Is a protocol log needed to diagnose comment 4?
Flags: needinfo?
First diagnosis:
My "Thunderbird".app seems to invoke a "thunderbird" process from time to time, consuming up to 97% CPU power. Killing it does lower the CPU power, and Thunderbird seems to work just fine anyhow.
Flags: needinfo?
I get two thunderbird IMAP protocol logs with Thunderbird 17.

* thunderbird_20121204-02.log
  I received a mail with a corrupted attached file.

* thunderbird_20121204-03.log
  When I restarted Thunderbird once, I got a correct attached file.

(a part of thunderbird_20121204-02.log)
5180[c384820]: FetchTryChunking: curFetchSize 11600919
5180[c384820]: FetchMessage peek: curFetchSize 11600919 numBytes 393216
5180[c384820]: d534400:imap-d.ecc.u-tokyo.ac.jp:S-INBOX:SendData: 11 UID fetch 31632 (UID RFC822.SIZE BODY.PEEK[]<0.393216>)

Thunderbird send a IMAP command to get only 393216 byte of the message(11599659 Byte).
(393216 = 0x60000?)
Maybe I received only part of the attached file.

(a part of thunderbird_20121204-03.log)
1860[10487820]: FetchTryChunking: curFetchSize 11599659
1860[10487820]: d634400:imap-d.ecc.u-tokyo.ac.jp:S-INBOX:SendData: 15 UID fetch 31632 (BODY.PEEK[2])

Thunderbird send a IMAP command to get whole of the message.
I got a correct attached file.
So, I don't want to mix up various bugs, but at least my problem (TB consumes lots of CPU power) seems to be related to a second process "thunderbird" spawned after a while, constantly hooking up 100% CPU power, without actually doing anything, it seems.

As soon as I kill this process, all is back to normal, and Thunderbird is not affected, it seems.

Here's the output during the high load

floeffmac:~ floeff$ ps auxw | grep -i thunder
floeff           803  98,9  0,0  3562536   1892   ??  R     1:28pm  44:01.03 /Applications/Thunderbird.app/Contents/MacOS/thunderbird -psn_0_565386
floeff           645   0,0  7,3  3645580 306392   ??  R    12:18pm   6:12.60 /Applications/Thunderbird.app/Contents/MacOS/thunderbird -psn_0_565386
floeff           425   0,0  0,0  3560520   1784   ??  S    10:42am   0:00.00 /Applications/Thunderbird.app/Contents/MacOS/thunderbird -psn_0_163880
floeff           414   0,0  0,0  3606440   1192   ??  S    10:41am   0:00.00 /Applications/Thunderbird.app/Contents/MacOS/thunderbird -psn_0_163880

and here after killing the process

floeffmac:~ floeff$ ps auxw | grep -i thunder
floeff           645   0,4  7,2  3617144 302884   ??  S    12:18pm   6:16.16 /Applications/Thunderbird.app/Contents/MacOS/thunderbird -psn_0_565386
floeff           425   0,0  0,0  3560520   1784   ??  S    10:42am   0:00.00 /Applications/Thunderbird.app/Contents/MacOS/thunderbird -psn_0_163880
floeff           414   0,0  0,0  3606440   1192   ??  S    10:41am   0:00.00 /Applications/Thunderbird.app/Contents/MacOS/thunderbird -psn_0_163880
Hi. I created test account thundertest@bk.ru , I also can publish a password if required.
If each of you could send there a couple of messages with no meaning, you all help me a lot with filling it. Attachments of 100Kb - 1Mb are welcome, but not in each message.
I recently got this or similar issue (bug 818831) reproduced after creating new mail folder through web interface of mailbox, but my mailbox already has up to thousand messages, some with attachments. Not sure that almost empty one will show the same results.
Thanks.

P.S. if someone has already filled test account, please try to reproduce bug by creating new folder in it, so that I need not to fill a new mailbox.
I've gathered some logs that may have sense.
Can't reproduce bug by creating/deleting folder in my own mailbox, but I made Thunderbird reload one already loaded folder. It reloads 99 messages from "Себе" folder, as it was doing when all database was being reloaded.
Logs are numbered in the order of executions, no execution happened between logs.
Launch 4 - folder "Себе" is valid.
Launch 5 - folder "Себе" is valid. Updating 1 message in folder "Вельд" stucks, windows shows no expected incoming internet traffic, whereas internet connection works and bk.ru server works too.
Launch 6 - folder "Себе" is being reloaded, updating 99 messages. No new messages were added to this folder in a whole series of launches.
In the next launches folder "Себе" continues being reloaded, reloading only remaining messages. Say, if Launch 6 loaded 13/99, Launch 7 will load 86 messages etc.
Also I tried to delete message from folder "Вельд" a couple of times somewhere between launches 3 and 9 (blame me, I don't remember exactly). Since task monitor displays "1 message is deleted from folder ...", at the next launch message is here again. May be it makes sense too, but I don't link these issues.

Logs are at ftp://insane-poet.ru/Misc/ThunderLogs.rar
Password to archive is product name in uppercase, followed by current major version without spaces. Feel free to PM me if my assistance is necessary.
(In reply to Makoto Itani from comment #8)

> Thunderbird send a IMAP command to get only 393216 byte of the
> message(11599659 Byte).
> (393216 = 0x60000?)
> Maybe I received only part of the attached file.

"mail.imap.chunk_size" parameter is this number (393216) on my Thunderbird.
 And, when I set "mail.server.default.fetch_by_chunks" parameter to "false", I can get correct attached file every time with IMAP.
A quick follow-up from my side:

The mail problem occurs rarely, but sometimes TB at least loads for quite a while on messages it already seems to have, but it's much better than with TB 16.

Independent from that, after a while, my TB 17 gets slow and unresponsive, and it's always a process called "thunderbird" (in comparison to "Thunderbird.app") consuming 70-100% of CPU cycles. If I kill it, all is well again. Seems TB forks itself for some reason?
(In reply to Florian Effenberger from comment #13)
> A quick follow-up from my side:
> 
> The mail problem occurs rarely, but sometimes TB at least loads for quite a
> while on messages it already seems to have, but it's much better than with
> TB 16.

FTR this is due to bug 806760 having been fixed in TB17.

For what's left (like bug 806760 comment 92), could you attach a file with your  protocol log please using
  imap:5,imapautosync:5,timestamp
see https://wiki.mozilla.org/MailNews:Logging


> Independent from that, after a while, my TB 17 gets slow and unresponsive,
> and it's always a process called "thunderbird" (in comparison to
> "Thunderbird.app") consuming 70-100% of CPU cycles. If I kill it, all is
> well again. Seems TB forks itself for some reason?

You never posted which addon was causing your CPU strangeness
Flags: needinfo?(floeff)
(In reply to Wayne Mery (:wsmwk) from comment #14)
> > Independent from that, after a while, my TB 17 gets slow and unresponsive,
> > and it's always a process called "thunderbird" (in comparison to
> > "Thunderbird.app") consuming 70-100% of CPU cycles. If I kill it, all is
> > well again. Seems TB forks itself for some reason?
> 
> You never posted which addon was causing your CPU strangeness

Florian, another thing to check per irving ... "restart, run a little while, send self a new message and click on notification, see if process starts to spin." and if it does, please get a stack dump of the spinning process.
Flags: needinfo?
Sunil, can you also do a protocol log?
Flags: needinfo? → needinfo?(funtoos)
(In reply to Wayne Mery (:wsmwk) from comment #17)

Thank you very much.
I wrote a comment to Bug 434054.
Sorry, I never came back to update. I could not provide the logs because I have been off TB for some time now. As soon as a stable release comes out, I will be giving it another try. I managed to **** off my IT really bad because official supported config is Windows with outlook and they found me out running Linux and TB because of the overload I created (I have about 100k messages in one folder, 25k in another).
Flags: needinfo?(funtoos)
Sorry for the insanely long delay.

So, it seems that at the moment, my problems have vanished - I just have no clue due to what, since it happened before the 17.0.2 upgrade already.
The high load has been caused by the Enigmail plugin in conjunction with Pinentry for Mac, it seems. As soon as I disabled it, all was well. However, now everything seems to work fine, even with Enigmail enabled.

I'll keep looking at this issue, and as soon as I find some traceable issues, I'll get back to you.
Flags: needinfo?(floeff)
Blocks: 798013
Anyone with this problem: If you are running Thunderbird >= 17 and you are using Gmail IMAP, have a look at Bug 816327.

If everyone meets these criteria, we should close this bug as a dup. If anyone does not meet these criteria, please comment so we don't close this bug.
per comment 21
Status: UNCONFIRMED → RESOLVED
Closed: 10 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: