426351 - Downloading stops, messages received since last startup are duplicated in Inbox, or crash [@ nsMsgLocalMailFolder::EndMove(int)]

Status: RESOLVED WORKSFORME

(MailNews Core :: Backend, defect)

Description

[Dan Kegel]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.13) Gecko/20080325 Ubuntu/7.10 (gutsy) Mozilla/5.0  (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051107 Firefox/1.5
Build Identifier: 2.0.0.12 (20080213)

If I leave Firefox open to Netvibes.com *or* leave Acrobat Reader open
(at least, we think those are the triggers), Thunderbird exhibits the 
following problem:

After Thunderbird has been open for a long time (days), eventually automatic 
download of messages stops.
Restarting Thunderbird gets message download working again, but 
something horrible has happened: all the messages downloaded and filed 
in folders during the previous Thunderbird session are duplicated in the 
Inbox!  This is horrible because it takes several hours to manually
clean out the duplicates.

If I compact folders nightly, the consequence of this problem is minimized,
but that is not operationally feasible since a) compacting folders takes over
an hour, and b) if I compact folders while online very bad things happen,
so I have to go offline first; that means that if I do this overnight,
when I come back to the computer, I have to wait for an hour as it downloads
all the messages received since I went offline.


Reproducible: Always

Steps to Reproduce:
1. Leave Firefox logged in to netvibes.com *or* leave Acrobat Reader open
2. Leave Thunderbird open and downloading messages (several thousand/day)
3. Wait a few days
Actual Results:  
Messages stop downloading, and after restarting Thunderbird, all messages from
previous session that had been filed automatically into folders are now
duplicated in Inbox

Expected Results:  
Message retrieval should not stop, and messages should not be duplicated

This has been happening for years, through many versions
of Thunderbird.  It happens somewhere between twice a week and once a month.
(The problem is actually happening to my wife, not me.
I don't know why she puts up with it.)

I'm a developer and QA person myself, so I'd be happy to
switch on logging or run a debug build or whatever, just
point me to it.

Comment 1

[Dan Kegel]

Correction:
the problem doesn't happen *only* when Thunderbird has been open for a long time, but it is much more likely to happen if it's been open for a long time.  It has been known to happen twice in the same day, but that's not too common

Comment 2

[Wayne Mery (:wsmwk)]

Is the datastore you are compacting imap, pop or local? 
And how big is it both in terms of MB and # messages?

Backup *everything* using mozbackup, and then try 
 ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk/
Results?

Comment 3

[Dan Kegel]

Um, the account is imap, but the datastore seems to be local,
and large (19GB):

$ cd /cygdrive/c/Documents and Settings/Liz/Application Data/Thunderbird/Pro
files/rwuukilh.default/Mail/mail.zilf.org
$ du | tail -1
19554433        .
$ find . -name '*.msf' | wc -l
    395     
$ find . -type f -size +10M -ls | wc -l
    168    
$ find . -type f -size +100M -ls | wc -l
     30    
$ find . -type f -size +1000M | xargs ls -s
2052428 ./2006-06.sbd/SecondHandKidzClothes
1633740 ./2006-06.sbd/SecondhandKidzClothes2
1258704 ./Inbox.sbd/PeachHead2
1219064 ./Inbox.sbd/SecondhandKidzClothes2
$ ls -l 2006-06.sbd/SecondHandKidzClothes
-rwx------ 1 Liz None 2101684761 Oct 31  2006 2006-06.sbd/SecondHandKidzClothes

(Every couple of years I move Inbox.sbd to YYYY-MM.sbd
to keep the files from getting too big.)

(Who'd have thunk that a mailing list about second hand kids' clothes
could be so busy?)

Think mozbackup will handle a 19GB datastore?

Comment 4

[Dan Kegel]

Attachment: Crash report

Last night, coincidentally, it crashed (as opposed to locked up).
Here's the crash report.

Comment 5

[Wayne Mery (:wsmwk)]

Need your crash id http://kb.mozillazine.org/Talkback#Getting_an_incident_ID

Comment 6

[Dan Kegel]

Here 'tis: TB44309934W
Don't know if it's related; the usual symptom is a hang.  But who knows.

Comment 7

[Wayne Mery (:wsmwk)]

(In reply to comment #3)
> Um, the account is imap, but the datastore seems to be local,
>...
> (Every couple of years I move Inbox.sbd to YYYY-MM.sbd
> to keep the files from getting too big.)

so a) you have subfolders in inbox? b) messages get filtered into them? c) is filter thunderbird or server side? d) how many messages in inbox?  e) in the subfolder(s)?

mozbackup is windows-only, so you'll have to copy the profile folder or use some other utility.  what happens with trunk?


Your thunderbird2 crash nsMsgLocalMailFolder::EndMove may be unrelated  (but keeping here for now). What was your response to Dialog "Do you want to compress?" 

TB44309934W
Just sitting overnight. Dialog "Do you want to compress?" was up. 
nsMsgLocalMailFolder::EndMove  [mozilla/mailnews/local/src/nsLocalMailFolder.cpp, line 2853]
nsCopyMessageStreamListener::EndCopy  [mozilla/mailnews/base/src/nsCopyMessageStreamListener.cpp, line 183]
nsCopyMessageStreamListener::OnStopRequest  [mozilla/mailnews/base/src/nsCopyMessageStreamListener.cpp, line 195]
nsMailboxProtocol::OnStopRequest  [mozilla/mailnews/local/src/nsMailboxProtocol.cpp, line 395]
nsInputStreamPump::OnStateStop  [mozilla/netwerk/base/src/nsInputStreamPump.cpp, line 564]
0x06201690
nsMailboxProtocol::`vftable'
PendingPACQuery::AddRef  [mozilla/netwerk/base/src/nsPACMan.cpp, line 102]
0x8b560c45

comments from other crashes with this top of stack
- sending a mail, it was sent OK but did not want to store a copy in the send folder as usual....several times. TB44294001
- bitdefender was analysing the emails TB42036500
- downloading emails TB41681531
- just pressed j in order to remove a junk e-mail from my inbox. TB40720688

crash a duplicate of bug 352814 or bug 371027?

Comment 8

[Dan Kegel]

> so a) you have subfolders in inbox?
Yes.

> b) messages get filtered into them? 
Yes.

c) is filter thunderbird or server side?
Thunderbird.

> d) how many messages in inbox?  
Today, Inbox is 100MB itself, and has 5000 messages in it.

> e) in the subfolder(s)?
About 18 gigabytes' worth across about 400 folders (see the distribution above).

> mozbackup is windows-only, so you'll have to copy the profile folder 
I'm running Windows (but I use cygwin for diagnosing problems),
so mozbackup should be ok; I haven't tried it yet.

> what happens with trunk?
Haven't tried yet.  Tempted to wait a week or two, we're about
to buy a new, 3x faster computer, partly to eliminate the possibility
of this bug being hardware-related.

> What was your response to Dialog "Do you want to compress?" 
"No."

Comment 9

[Wayne Mery (:wsmwk)]

(1yr update)

Dan, do you still see this problem in TB2?
If you do, can you try beta?
 http://www.mozillamessaging.com/en-US/thunderbird/early_releases/
 (backup profile first)

Comment 10

[Dan Kegel]

We have not yet tried TB2 (wife is risk-averse).  It still happens with
our new fast hardware.  We will try the TB2 beta when we get a chance.

Comment 11

[Wayne Mery (:wsmwk)]

I'd think this is the same as rare trunk crashes nsMsgLocalMailFolder::EndMove(int)

FWIW, I can't find anything more recent than 3.0b3 bp-8c44abcb-3350-459f-8fd5-e5f3a2090828 -- might be gone, might not.

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to comment #10)
> We have not yet tried TB2 (wife is risk-averse).  It still happens with
> our new fast hardware.  We will try the TB2 beta when we get a chance.

TB2 = version 2. comment 9 was referring to two versions, v2, and v3(beta form)

I rechecked I don't see any crashes newer than 3.0b2, which is now ~5 months old. So I'm going to declare the crash is gone based on examination of crash-stats.mozilla.com.  If you still see the problem when version 3 becomes available then please reopen the bug. 

i would say, optimistically, your second issue of duplicate messages has a good chance of also being gone.  But feel free to open a new bug on the non-crash issues.  Logging is accomplished with information at https://wiki.mozilla.org/MailNews:Logging

Comment 13

[Dan Kegel]

Thanks, I will.  TB crashed again on this last night, interfering with
my domestic tranqulity :-)  Just need to work up the energy to do
a backup, then we can try the RC (or final).