Closed Bug 482836 Opened 15 years ago Closed 8 years ago

Multiple (iterations) copies of draft messages are saved in the DRAFTS folder (local Drafts folder, not IMAP). Obsolete drafts should be deleted

Categories

(Thunderbird :: Folder and Message Lists, defect)

Product:
Thunderbird
Component:
Folder and Message Lists
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1216914

People

(Reporter: davofanmail, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: dogfood, qawanted, testcase-wanted) 

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)
Build Identifier: Eudora 8.0.0b5 which is T'Bird Pre1

Several copies (of increasing length) of a draft message are kept in the Drafts folder, even after the message has been fully composed and sent.

This is a regression from previous releases.

Reproducible: Always

Steps to Reproduce:
1.Write an email
2.Take a long time doing so
3.
Actual Results:  
Several copies of the draft remain in the DRAFT folder, even after the mail has been sent.

Expected Results:  
Only one copy of the draft message at all times, and once it's been sent it's expunged.
Assignee: nobody → mozilla-bugs
Product: Thunderbird → Penelope
QA Contact: general → general
Not sure if it is always reproducible, but I often do find multiple copies of my drafts in Drafts.
This is quite reproducible if you use IMAP and set your account to mark messages as read when deleted instead of having them moved to a trash mailbox when deleted. 

- Set Eudora to mark deleted messages as deleted (not to move to a Trash mailbox)
- New message
- Enter address, subject and some text in body
- Save
- Enter more text in body
- Save
- Wash, rinse, repeat
- Look in Drafts folder
- Note multiple copies of your message
Status: UNCONFIRMED → NEW
Ever confirmed: true
FYI this happens to me about 50% of the time when I am editing messages for a long period (say, 30 minutes).  But I'm totally a POP user.

I've been using Eudora since the early betas and this never happened before Beta 5.
I see this same behavior in Thunderbird 3.0b2 so I am moving it there.

Matt
Assignee: mozilla-bugs → nobody
Component: General → Mail Window Front End
OS: Windows XP → All
Product: Penelope → Thunderbird
QA Contact: general → front-end
Hardware: x86 → All
Target Milestone: --- → Thunderbird 3.0b2
Version: unspecified → Trunk
 (In reply to comment #0)
> Reproducible: Always
(In reply to comment #3)
> FYI this happens to me about 50% of the time when I am editing messages for a
> long period (say, 30 minutes).

"Always" usually means 100% in natural languages. David Taber, which? 100%? 50%?

(In reply to comment #0)
> 2.Take a long time doing so
(In reply to comment #3)
> FYI this happens to me about 50% of the time when I am editing messages for a
> long period (say, 30 minutes).

Do you use auto-compact with mail.purge.ask=false?

If yes, change your setting.
 - Account Settings/Copies & Folders/
   Check "Show confirmation dialog when messages is saved"
 - mail.purge.ask=true
And If dialog for auto-save appeared while composing, CANCEL it.
(never check "Do it automatically..." at the dialog) 
Can you reproduce the problem?

(In reply to comment #0)
> This is a regression from previous releases.
(In reply to comment #3)
> I've been using Eudora since the early betas and this never happened before Beta 5.

Really regression?

Phenomenon of "remaining draft mail" could be observed by next test.
(observed with Tb trunk and Seamonkey 1.1.14 on MS Win-XP) 
 0. Display "Order Received" column of local Drafts(offset in mail folder file)
 1. Copy several mails to Drafts (say mail-A)
 2. "Save As Draft" at compose window
 3. Copy several mails to Drafts (say mail-B)
 4. Delete a mail in mail-A
 5. Compact Folder of "Drafts" (offset of draft at step 2 is changed)
 6. "Save As Draft" at compose window
    => draft at step 2 still remain 

Default of auto-compact/auto-save related settings was changed by Beta 5, wasn't it? (Default is not written in prefs.js. Check it by "Config Editor")
(In reply to comment #4)
> I see this same behavior in Thunderbird 3.0b2 (snip)

Matt Dudziak, your Bug 426373 is involved in the "same behaviour"? (Drafts=Sent case)
Correction. Sorry for spam.
(Wrong) And If dialog for auto-SAVE appeared while composing, CANCEL it.
=> And If dialog for auto-COMPACT appeared while composing, CANCEL it.
When confirmation after auto-SAVE or "Save" appeared, reply OK, please.
(In reply to comment #5)
>  (In reply to comment #0)
> > Reproducible: Always
> (In reply to comment #3)
> > FYI this happens to me about 50% of the time when I am editing messages for a
> > long period (say, 30 minutes).
> 
> "Always" usually means 100% in natural languages. David Taber, which? 100%?
> 50%?

Ah, I only use supernatural languages, so it's easy to misunderstand (he says, with a Schroedinger's Cheshire smile).  Check out quantum mechanics if you want to know more.

But anyway, sorry for my imprecision...I don't look at the draft box often enough to see if this is happening all the time, or only on some messages.  The longer I keep the edit window open, the more likely it is I'll have several copies of the message there.

> 
> (In reply to comment #0)
> > 2.Take a long time doing so
> (In reply to comment #3)
> > FYI this happens to me about 50% of the time when I am editing messages for a
> > long period (say, 30 minutes).
> 
> Do you use auto-compact with mail.purge.ask=false?
> 
> If yes, change your setting.

YES, I was using that setting.  I have changed it to TRUE.  Let's see if the behavior stops.

>  - Account Settings/Copies & Folders/
>    Check "Show confirmation dialog when messages is saved"
>  - mail.purge.ask=true
> And If dialog for auto-save appeared while composing, CANCEL it.
> (never check "Do it automatically..." at the dialog) 
> Can you reproduce the problem?
>
I will look at my draft folder and see if the problem continues over the next couple of days. 


> (In reply to comment #0)
> > This is a regression from previous releases.
> (In reply to comment #3)
> > I've been using Eudora since the early betas and this never happened before Beta 5.
> 
> Really regression?

It's a new behavior.  Perhaps it's not a regression, but simply a new side-effect of a different default configuration in newer release.


> 
> Phenomenon of "remaining draft mail" could be observed by next test.
> (observed with Tb trunk and Seamonkey 1.1.14 on MS Win-XP) 
>  0. Display "Order Received" column of local Drafts(offset in mail folder file)
>  1. Copy several mails to Drafts (say mail-A)
>  2. "Save As Draft" at compose window
>  3. Copy several mails to Drafts (say mail-B)
>  4. Delete a mail in mail-A
>  5. Compact Folder of "Drafts" (offset of draft at step 2 is changed)
>  6. "Save As Draft" at compose window
>     => draft at step 2 still remain 
> 
> Default of auto-compact/auto-save related settings was changed by Beta 5,
> wasn't it? (Default is not written in prefs.js. Check it by "Config Editor")

hmmmmm, I don't see any variables/attributes that look like auto-compact or auto-save in the Config Editor.  What string should I be searching for?
(In reply to Comment #8)

Check following entries, please.
>(auto compact)
> mail.prompt_purge_threshhold : true = auto-compact is enabled
> mail.purge_threshhold : NNNN (in KB, Kilo bytes)
> mail.purge.ask : true  = Display confirmation dialog before auto compact
>                  false = Do auto compact silently
>(auto save)
> mail.compose.autosave : true = auto-save is enabled
> mail.compose.autosaveinterval : NN (in minutes)

(Off-Topic)

> (he says, with a Schroedinger's Cheshire smile)
Roger. I've recalled that I had barely got course credit of "quantum chemistry" in my student days, without any cheating :-)

threshhold is used in prefs.js setting instead of threshold.
(See http://en.wiktionary.org/wiki/threshhold for "threshhold")
It's perhaps the reason why you couldn't find related settings.
(In reply to comment #9)
> (In reply to Comment #8)
> 
> Check following entries, please.
> >(auto compact)
> > mail.prompt_purge_threshhold : true = auto-compact is enabled

YES, was set to TRUE

> > mail.purge_threshhold : NNNN (in KB, Kilo bytes)

Was set to 500

> > mail.purge.ask : true  = Display confirmation dialog before auto compact
> >                  false = Do auto compact silently

Was set to TRUE


> >(auto save)
> > mail.compose.autosave : true = auto-save is enabled
> > mail.compose.autosaveinterval : NN (in minutes)

this was "enabled" and "5 mins"
> 

So, these were all set the way they should have been.

My test of "is this problem continuing" is to have two mails open in draft form, adding a few words every 10 minutes or so.

So far, here's the behavior:
  - a message that I have close-saved and re-opened is in the draft box,
    only one copy
  - a message that I have just kept open and have never saved is NOT in the
    draft box, and will presumably be lost on a crash.

This is the behavior I expect, I'll test for a few hours.  IF I DON'T REPORT BACK HERE within the next 24 hours, assume that the system is behaving correctly and (I guess) this bug can be closed.
I'm baaa-aack!

Well, with all the settings as described above, here's the behavior...which I believe is a bug and is 100% reproducible.

As long as T'Bird stays up, it doesn't seem to create multiple copies of open outbound drafts.  So far, so good.

But if you shut T'Bird down, it will create near-clones of the draft messages.  It seems to only create two, but it seems to do it every time there's a shutdown/restart cycle.
(In reply to comment #10)
>(snip)
> So far, here's the behavior:
>(snip)
> This is the behavior I expect, (snip)
(In reply to comment #11)
>(snip), here's the behavior...which I believe is a bug and is 100% reproducible.

"Tb's behavior you expected" == "the behavior which you believe is a bug" ?
Or your "the behavior which you believe is a bug" in Comment #11 == "Tb's behavior after compact of Drafts folder" which I wrote in my Comment #5 ?
Again, sorry for speaking in supernatural language.

Behavior in #10 is not the same as #11.

Tb is creating phantom clones of draft messages when the messages have been edited during a Tb session, but not sent, and then Tb is shut down/restarted.

What I would expect is that a draft message only shows up once in its folder, no matter how much or under what conditions it has been edited.
(In reply to comment #13)

Which?
(A-1) "Compaction of Drafts folder during composing/draft saving"
      is irrelevant to your problem.
(A-2) "Compaction of Drafts folder during composing/draft saving"
      is relevant to your problem.

Which?
(B-1) "auto-save" is irrelevant to your problem.
(B-2) "auto-save" is relevant to your problem.
If my memory is not corrupted, I saw next phenomenon in the past.
   "Save As" => first draft is saved.
   By auto-save, second draft is created.
   After "Send"/"Send Later", no draft mail remains.
   If compose window is closed without send operation, only single copy is kept.

Check "Drafts" file content after each operation which relates to "saving draft" and "deleting of previous draft". "delete of previous draft" is done by setting MSG_FLAG_EXPUNGED bit=ON in X-Mozilla-Status: header of mail data for "previous draft".
( See http://www.eyrich-net.org/mozilla/X-Mozilla-Status.html?en )
A-1 vs A-2:  unknown.  I am not intentionally triggering compaction of the draft folder, and have raised the compaction auto-trigger threshold to 5 MB so it doesn't happen as often.  Suspect A-1, but don't know how I'd prove it.  

B-1 vs B-2:  suspect B-2, but have no hard data.  Auto-save is occurring, and has been for several versions of Tbird.  As stated earlier, my issue of clones  is a "new" phenomenon, where I haven't changed my behavior but Tbird has.  

I just made a quick test of the issue, checking the MSG_FLAG_EXPUNGED bit.  Naturally, I couldn't reproduce the problem (so much for my 100%), but the bit was never set no matter what the message state was in the drafts folder.  Will keep trying to make it happen and report back on the bit status.
(In reply to comment #15)
> A-1 vs A-2:  unknown. (snip) Suspect A-1, but don't know how I'd prove it.
> B-1 vs B-2:  suspect B-2, but have no hard data. 
> Auto-save is occurring, and has been for several versions of Tbird.
> As stated earlier, my issue of clones is a "new" phenomenon, where I haven't changed my behavior but Tbird has.  

Question to clarify.
When the "your issue of clone/new phenomenon" occurred,
  - Was confirmation dialog for auto-compact displated by Tb?
    (mail.purge.ask=true is to force display of this dialog)
    If yes, did you reply "CANCEL"?
    Still mail.purge.ask=true?
  - Was dialog after auto-save displayed by Tb?
    Check of "Show confirmation dialog when messages is saved" is for this.
    mail.identity.idN.showSaveMsgDlg=true/false is corresponding setting.
to answer from previous comments:
   - when the dupe messages occur in the drafts folder, their mozilla status word is "0001" on both copies of the message
   - confirmation of auto-compact has been occurring.  however, it hasn't occurred yet on the draft folder due to 5 MB threshold.  on other folders (particularly in-box), I have been allowing the auto-compact to occur (assuming it's only compacting that folder).
   - mail.purge.ask=true
   - I don't see the mail.identity.idN.showSaveMsgDlg variable in prefs.js or config editor.  So now I have added this to prefs.js setting the true value on all accounts.
   - I didn't mention that I have turned Tbird's internal indexing off.  I have no other indexer on this machine.
   
Just to go ambiguous on you, the false cloning of draft messages is happening, but it is certainly not 100% repeatable.  I haven't figured out what the conditions are that trigger it, and I can sometimes go several hours without it occurring even though I have a couple of messages open as drafts.
whoops:  just discovered that when I accept the dialog to compact any folder, Tbird may compact drafts as well.  

So I have been inadvertently allowing drafts to be compacted up to now.  I will hit cancel from here forward, to assist with the test.
Another case was found.
(A) Outdated .msf file (mismatch between mail folder file and msf file).
    (e.g. crash/freeze while appending mail data to Drafts file)
    Note: This condition is generated by;
      - Terminate Tb/Eudora/Seamonkey.
      - Copy mail data of deleted draft mail(X-Mozilla-Status: 0009 or 0008)
        multiple times in Drafts file.
        => File size of Drafts file is increased. 
(B) Saving of draft is executed before explicite open of Drafts folder.
    (Draft folder is not clicked yet at folder pane)
If (A) & (B) occurs at same time, drafts mail data is normally appended to Drafts file by save operation, but old draft mail is not deleted upon next save.
(MSG_FLAG_EXPUNGED bit in X-Mozilla-Status: header is not set.)
So, when Drafts folder is opened by folder click, internal re-build index is invoked due to (A), then undeleted old draft mails appear in Drafts folder as active draft mail.

I think above is remaining issue around outdated .msf handling, even after fix of Bug 472446, Bug 471307, Bug 471682.
additional info about my symptom (which I've now managed to create without a stop/restart cycle in Tbird).

Once the symptom has started (where there are clones of a draft message), the cloning message will reproduce multiple times even though some other open draft messages will not reproduce.  Here's the scenario:
  - create 4 distinct messages, put some content in them
  - leave them open, add some new distinct content every hour
  - some of the drafts will be fine, no symptom
  - other draft messages will be cloned (seems to be occurring at the auto-save time)
  - once a draft message is cloned, the original message is more likely to clone itself repeatedly (note, I continue to add new content to the original set of messages every hour)

Please advise what useful testing I can perform.  Don't have too much time to waste ;-)
Whoops, pushed "commit" too soon.

Once messages have been cloned, they can't be deleted from the draft folder.  Or at least they pretend not to be deleted.  Using ctrl-D or drag/drop to the trash folder, the messages remain in the draft folder until I terminate and restart Tbird.

On restart, the messages behave more or less normally.
(In reply to comment #20)
> Here's the scenario:
>(snip)
>  - once a draft message is cloned, the original message is more likely to
>    clone itself repeatedly

I couldn't/can't see such phenomenon by "compaction of Drafts while composing", with any of Sm 1.1.15, Tb 2.0.0.21, Tb latest-trunk. I could/can see only "single undeleted draft mail per a compact folder during composition". I could see such phenomenon(multiple undeleted old draft mails for single composing mail) by test in Comment #19 only, without "compaction of Drafts during composition".

Do step-by-step check of followings.
 - Timestamp & file size of "Drafts" & "Drafts.msf" file
 - MSG_FLAG_EXPUNGED bit after each "Save As Draft" or auto-save
 - Mail count(unread & total) of Drafts at folder pane
   If old draft is normally deleted, count display becomes as follows.
     first save => unread count=1, second save => unread count=2
     => dialog after save => OK => unread count returns to 1
 - Timestamp & file size of "nstmp" & "nstmp.msf" file during compaction
 - When Drafts folder content check, display "Order Receievd" column(offset)
FYI.
If "Save As Drafts"/auto-save is executed during Drafts compaction is in progress, phenomenon of Bug 485348 Comment #3 occurs. See also Bug 485348 Comment #5(double draft backups in "Sent").
(In addition to comment #22)
Because activity on files has relation to issue, monitoring of it will also help your problem analysis.
- Start Process Monitor. Track I/O by Tb to Drafts, Drafts.msf, nstmp, nstmp.msf.
> http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
> http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
I've met third case.
With Options/"Send a copy to" of compose window, I could observe phenomenon of "multiple saved draft" and "undeleted old draft" at same time.

(test)
- Folder for draft : Copies&Folders = Draft-A, Options/Send a copy to = Draft-B
- Compose a mail, repeat save as draft, with changing subject upon each save.
(result)
- Dialog says "Saved to Draft-B" (folder of Options/Send a copy to)
- Draft mail is saved in both Draft-A and Draft-B.
  If "Send a copy to" == Draft-A, double draft mail is created in Draft-A
- Old draft mail in any draft folder is not deleted upon save as draft
Above phenomenon was observed with both Tb 2.0.0.21 and Tb trunk(2009/3/31 build).

David Taber(bug opener), do you use "Send a copy to" option sometime?
Hi WADA -

you asked do I use a "Send a copy to" option -- the answer is no.
I occasionally use the Cc: line within the actual mail, but that has nothing to do with the draft message cloning.

Also made discovery -- having mail indexing on makes the symptom occur much more often (I'd had it turned off recently, due to another bug I was running into).  If a draft message is open *but unedited* it won't be cloned.  If it's open and being edited, a clone will be created every 5 minutes while editing is in progress.  

Probably some interaction between auto-save and indexing thread...
FYI. I've opened Bug 488075 for issue of Comment #25("Tools/Send a copy to" issue).
Seeing this too. As example I've open draft do very little changes close & save it. Go into drafts folder there two drafts while others are deleted (I'm using mark as deleted scheme).
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b5pre) Gecko/20090428 Shredder/3.0b3pre
(In reply to comment #28)

Although common issues with local mail folder case(e.g. interfere by auto-compact/indexing etc., which we are suspecting in this local mail folder case) can exists in similar external symptom of IMAP case, IMAP case is affected by interaction with IMAP server and Tb's option setting such as "Delete Model", "auotomatic expunge" etc., and IMAP particular issue is involved in similar external symptom in many cases. (for example, bug 462880, bug 490556)
Nikolay Shopik, get IMAP log and check IMAP level flow first, and open separate bug for your problem, referring to this bug, please. If common issue with this bug is found by your analysis, please let us to know.
(In reply to comment #29)

> Nikolay Shopik, get IMAP log and check IMAP level flow first, and open separate
> bug for your problem, referring to this bug, please. If common issue with this
> bug is found by your analysis, please let us to know.

I'm pretty sure you know where to find the instructions for getting the logs - but just in case they are at : https://wiki.mozilla.org/MailNews:Logging
what auto save interval value?
sometimes I see this when I set low, like to 1 minute
elevating to major+dogfood, as it is becoming increasingly difficult to trust the contents of drafts folder.  (I'm sure I wrote that in another bug somewhere, but I'm not sure where)  So, would be nice to be permanently rid of this issue!

I wonder if  Bug 279366 and bug 382517 (and perhaps even bug 78391) offer any hints as to what is happening
Severity: normal → major
Component: Mail Window Front End → Folder and Message Lists
Keywords: dogfood, qawanted
QA Contact: front-end → folders-message-lists
Whiteboard: [needs log]
see also bug 460085
(In reply to comment #32)
> elevating to major+dogfood, as it is becoming increasingly difficult to trust
> the contents of drafts folder.  (I'm sure I wrote that in another bug
> somewhere, but I'm not sure where)  

OK, that was in bug 307046, which fixed a problem in this area over 2 years ago.

Jim writes something similar in bug 382517 comment 25
Flags: blocking-thunderbird3?
Adding "local, not IMAP" to bug summary, to avoid confusion.
Summary: Multiple (iterations) copies of draft messages are saved in the DRAFTS folder → Multiple (iterations) copies of draft messages are saved in the DRAFTS folder (local Drafts folder, not IMAP)
I don't think we can block on this. There are no clear steps to repeat and the frequency doesn't seem that great. I've been using local folders for drafts for a while and haven't seen an issue either (though I'm not saying the issue isn't here).
Flags: wanted-thunderbird3+
Flags: blocking-thunderbird3?
Flags: blocking-thunderbird3-
Keywords: testcase-wanted
Whiteboard: [needs log]
Target Milestone: Thunderbird 3.0b2 → ---
PMFJI, but I am seeing similar behavior on:

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.8.1.22) Gecko/20090704 SeaMonkey/1.1.17

I hadn't noticed until recently how many drafts wewre accumulating. I do *not* see, however, the symptoms described in comment #20 and comment #21. I simply have a number of saved drafts (drafts are configured to save every 2 minutes). I have only noticed this with my POP3 account, though admittedly, I don't use IMAP all that much on this particular box (I will test, however, to see if this may be related to bug 460085.

This behavior did not occur under SeaMonkey 1.1.6.

I'll keep an eye on the situation and will report any additional behavior which may appear relevant (no need to bug spam, I'm sure we'll all agree).

Cheers.

Lewis
I am no longer seeing this (note my comment #37) with:

Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.9.1.16) Gecko/20101127 Lightning/1.0b2 Mnenhy/0.8.3 SeaMonkey/2.0.11

I'm now using about 50/50 POP3/IMAP, and I do not see it for either type of account. I will follow up on bug 460085 separately.

(Wayne, thanks for the reminder. ;-)

Cheers.

Lewis
Nikolay, do you still see this?  And are your settings similar to comment 2 testcase?

(reporter David seems to be gone)
I'm using "mark as deleted" model, so if multiply copies of marked as deleted messages count as this case then yes i'm seeing this, but I doubt this is real problem in "mark as deleted" model. In this mode TB doesn't purge message, thus there will be always multiply copies of drafts, especially when you composing large message or just saving a lot.
:bienvenu, while writing "compose in a tab", I've hit a difficulty, and I think it might be related to what's wrong here. What I'm doing is exactly what http://mxr.mozilla.org/comm-central/source/mail/components/compose/content/MsgComposeCommands.js#2756 is doing, except the original file has silent failures... while my code doesn't. That might explain why no one has found anything meaningful in the error console so for.

Long story short, it looks like the code in there is doing something hackish... but I couldn't find a "cleaner" way of doing this. The draftUri starts with imap-message://... I don't know if this is a message URI or a Necko URL but I couldn't get the corresponding message header without using the same string manipulation stuff that's in MsgComposeCommands.js.

Anyway, when calling folder.GetMessageHeader(msgKey), I get

Error: uncaught exception: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIMsgFolder.GetMessageHeader]" nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)" location: "JS frame :: chrome://kompose/content/stub.ui.js :: <TOP_LEVEL> :: line 102" data: no]

why might give a hint as to why it's failing... thoughts?
I also forgot to say that this happens randomly for my code as well, the error doesn't always pop up. Maybe we could check-in a small patch that enables error logging in that part of the code...?
(In reply to comment #41)
> :bienvenu, while writing "compose in a tab", I've hit a difficulty, and I think
> it might be related to what's wrong here. What I'm doing is exactly what
> http://mxr.mozilla.org/comm-central/source/mail/components/compose/content/MsgComposeCommands.js#2756
> is doing, except the original file has silent failures... while my code
> doesn't. That might explain why no one has found anything meaningful in the
> error console so for.

Last I spoke to bienvenu on this subject, he implied that bug 629208 could actually cause this. I'm not totally convinced yet, as I wonder if we're not actually refreshing the drafts folder correctly when we save a draft to it.
Blocks: 618553
we have a mess of drafts bugs, and with my current problem I don't know which one to turn to. 

it would be very, very cool if someone could sort out:

a) a solid testcase for fixing
b) decide which are the "granddaddy" blocker bugs out of bug 382517, this bug, bug 460085 and perhaps others and alter summaries as needed to distinguish them. (it's not clear to me that we have dependencies correct - eg why does this block a gmail bug?) and then
c) put these drafts bugs [1] in the right buckets as duplicates. perhaps even some of the confirmed bugs are duplicates. 
d) with respect to FIXED Bug 629208 imap save as draft doesn't increment unread count of drafts folder - most of the list predates it and none filed after it are marked regression
e) fixes!

[1] multiple drafts bugs https://bugzilla.mozilla.org/buglist.cgi?list_id=2187801&short_desc=draft&field0-0-0=short_desc&bug_severity=major&bug_severity=normal&resolution=---&query_format=advanced&short_desc_type=allwordssubstr&type0-0-0=anywordssubstr&value0-0-0=delet%20duplic%20multip%20two&product=MailNews%20Core&product=Thunderbird
Summary: Multiple (iterations) copies of draft messages are saved in the DRAFTS folder (local Drafts folder, not IMAP) → Multiple (iterations) copies of draft messages are saved in the DRAFTS folder (local Drafts folder, not IMAP). Obsolete drafts should be deleted
(In reply to Wayne Mery (:wsmwk) from comment #44)
> a) a solid testcase for fixing

Following are solid or semi-solid cases which produce phenomenon of "undeleted old draft mail",
(i)   Comment #5, by Compact of local Drafts folder,
(ii)  Comment #19, by out-dated-msf condition of Local Drafts folder, 
(iii) Comment #25, by "Send a copy to" == Drafts,
and there is known issue which can produce phenomenon of "undeleted old draft mail";
(vi)  Contention between "new draft save and old draft delete" and "old draft delete by mail send",
but any of above doesn't look to applicable to bug opener's problem nor problem reported by other people in this bug.
And, if above case, phenoenon is usually "one old draft mail is not deleted" instead of "multiple old mails of a single composed mail are not deleted".

For (i), bug for it may already exist but I'm not sure.
For (vi), some bugs already exist.

If IMAP folder, actual "multiple old mails of a single composed mail are not deleted" can occur. For example, 
(v) Server doesn't support SEARCH for Message-ID: correctly
    then Tb can't delete old draft mail.
For "Unified Folder" view(or Search Folder/Virtual Folder) for IMAP Drafts folder with IMAP delete model of "Just mark it as deleted", phenomenon of faked "undeleted old draft mail" is known.
(vi) "Old draft mail with \Deleted flag" is not shown with strike-thru line
    if virtual folder, so it looks for user that any old draft mails are not
    deleted by Tb.
IIRC, (vi) is resolved by hiding "mail with \Deleted flag" at Unified Folder in recent release.

However, this bug is for problem on local Drafts folder since initial. So these IMAP related issues should be excluded from this bug.
No longer blocks: 618553
Blocks: tb-drafts
I can confirm this problem. It is also more or less identical to #402132.
I'm the original poster....and can confirm I still get this problem occasionally, four years later.  My config is POP only, Windows7 Pro 64 bit, TB 17.

Speculate that it may have something to do with an interaction with the windows file system, where locks can step on each other (explorer and the windows indexer are fairly famous for this).
I can confirm this problem on TB 24.0.1, Windows 7 32-bit working with IMAP account on Outlook.com

There will be several emails saved in the Drafts folder and they are not gone once the message is finally sent. Refreshing the folder still shows the emails there. I have to delete them manually.
I'm the guy who originally filed this bug....coming up on the five year anniversary!  Still occurs sometimes.
Configuration is now Win7 64 bit pro, TB 24.1.0 with TB indexing on.
This still occurs for me with Thunderbird 24.3.0.  It's been happening to me for years, I should have piled on to the bug report earlier.
Can everyone who has reported this bug indicate:
  * are you running Windows (and if so, which version)?
  * do you have windows indexing on?
  * do you have the Drafts folder in a Windows Encrypted File System or BitLocker encrypted directory?
  * do you have TB indexing (Glodia) turned on?
  * are you using POP or IMAP?

For me: win7pro 64 bit with indexing on, POP, glodia on, and the drafts folder in encrypted directory.

I suspect this has to do with conflicting locks (app vs service vs OS)
For me it is a problem on a Mac. However, Mail.app (the built in client in Mac OS X) also shows this behavior but not Mulberry (AFAIK).
(In reply to David Taber from comment #51)

Win 7 Pro 64-bit, Thunderbird 24.4.0, windows indexing on, no encryption on drafts folder or drive, POP. In Tools, Options, Advanced, General, Advanced Configuration, I have the following two items checked:
"Allow Windows Search to search messages" and
"Enable Global Search and Indexer".
In Tools, Options, Composition, General I Auto Save every 1 minute.

At least once a week I find an extra copy of a sent email in my Drafts folder. Usually it occurs on emails that I had open composing for an hour or more. Occasionally there are two or three extra copies left in Drafts. This minor problem has occurred for years.
(In reply to David Taber from comment #47)
> Speculate that it may have something to do with an interaction with the
> windows file system, where locks can step on each other (explorer and the
> windows indexer are fairly famous for this).

Please supply references regarding his information
Flags: needinfo?(davofanmail)
References, I don't got.
Experiences, I have plenty.
I've been using NTFS with the EFS and file system indexing feature for 10 years on XP, Vista, and Windows 7, all pro, all 64-bit.  If you run a heavy workload on the system for long periods, eventually one of the applications will not be able to save its file, because "another process has a lock on the file" (or some similar error message).  The app will be able to save the file under a new name, but the existing file has been locked by some (combination of) stuff in the system.  Sometimes, the symptom will go away after a few minutes (speculation: the fs indexer has done its stuff and moved on), and sometimes the only cure is a reboot (speculation: two or more locks are stepping on each other).
This definitely affects TB occasionally in other areas (i.e., having nothing to do with this bug report).  For example, I may be able to download mail in 5 of my accounts but the 6th one pretends to have nothing new even though a days' worth of messages are queued up in the POP server.  I can attempt getting new mail a dozen times, getting only an instant "nothing there"...but as soon as I kill TB and restart it all those mails come flying in.  If that doesn't work (maybe 20% of the time), I have to reboot and then all the mails appear.
If my speculation is right, TB is being piggish in its file locking (i.e., keeping the files it needs locked at all times, not just at data-integrity-critical intervals).  If that is the issue, then a solution strategy would be for TB to unlock all its files once an hour, wait a few seconds, and then re-lock them (although the details would obviously be more subtle / complex than that).
Flags: needinfo?(davofanmail)
David Taber, you have tried Unlocker? A simple Explorer extension.
DB--
Thanks for the pointer to unlocker...never heard of it, obviously haven't used it.
Will try it...but...the problem I'm having with the drafts folder files (and some other MSF issues) is happening at random (long) intervals, sometimes without any other symptoms.  So...by the time I notice the issue, the draft messages that "shouldn't be there" are actually several days old already.
So I'm not sure that unlocker is going to reveal very much.
(In reply to David Taber from comment #55)
> Experiences, I have plenty.

Do you understand my comment #5?
Why still no response like "your problem is different from comment #5" nor "your problem is same as comment #5" from you?
Because "Order Received" column value==messageKey and messageKey==messageOffset if local mail folder(except special exceptional case), "same as comment #5 or different from comment #5" is pretty easily known merely by watching "Order Received" column value of draft mails in Drafts folder, as long as a people can read number, a people can say which is larger/smaller/same, a people can remember number which was shown at screen once but disappeared after an action(save as draft), even if a people speaks ultra super natural language :-)

I created an addon for other bug such as bug 905576. In it, I added MsgDB Event Listener code, and added feature to log HdrAdded/HdrDeleted event. And my addon has feature to print MsgDBHdr object property such as messageKey/messageOffset in a mail folder.
Because "draft save" is merely set of "HdrAdded at Drafts folder for new version of draft mail" + "HdrDeleted at Drafts folder for previous version of draft" from perspective of messageKey/messageOfffset, what occurs is visible merely by getting log for HdrAdded/HdrDeleted events at Drafts folder.
If you want to see HdrAdded/HdrDeleted event log at your Drafts folder, let me know. I'll let you know uploaded site and how to use it.
Hi Wada,
I hadn't noticed comment #5 (from 4 years ago)...will put a reply to it in addition to this response to your latest comment.
I have not been viewing the Order Received column up to now, have turned that on.
It sounds as if your special add-on would provide some good diagnostic info, please do point me to it so I can provide extra info.
Thanks
David (my direct mail is davofanmail, if you look in the Cc: list)
(In reply to WADA from comment #5)
>  (In reply to comment #0)
> > Reproducible: Always
> (In reply to comment #3)
> > FYI this happens to me about 50% of the time when I am editing messages for a
> > long period (say, 30 minutes).
> 
> "Always" usually means 100% in natural languages. David Taber, which? 100%?
> 50%?

This used to occur "all the time" under certain conditions.  This is no longer true, given the change in OS and TB versions.  Now it is fairly rare, maybe once a month.  I have no idea what the triggering conditions are now.
> 
> (In reply to comment #0)
> > 2.Take a long time doing so
> (In reply to comment #3)
> > FYI this happens to me about 50% of the time when I am editing messages for a
> > long period (say, 30 minutes).
> 
> Do you use auto-compact with mail.purge.ask=false?

YES, .ask=false, .min_delay=480, .timer_interval=5, purge_threshold=1, purge_threshold_mb=100, purge_threshold_integrated=true
> 
> If yes, change your setting.
>  - Account Settings/Copies & Folders/
>    Check "Show confirmation dialog when messages is saved"
OK, checked this box in all 8 accounts

>  - mail.purge.ask=true
OK, changed it 

> And If dialog for auto-save appeared while composing, CANCEL it.
This implies that my draft will be wiped out if TB or the OS crashes while I'm composing.  Right?

> (never check "Do it automatically..." at the dialog) 
> Can you reproduce the problem?
Will let you know...

> 
> (In reply to comment #0)
> > This is a regression from previous releases.
> (In reply to comment #3)
> > I've been using Eudora since the early betas and this never happened before Beta 5.
> 
> Really regression?
I don't have a test lab and I don't have prior data so regression is indeed incorrect.  This is a mis-behavior that crept into TB as it and windows evolved.
(In reply to David Taber from comment #60)
> > And If dialog for auto-save appeared while composing, CANCEL it.
> This implies that my draft will be wiped out if TB or the OS crashes while I'm composing.  Right?

Sorry. It should be "Auto-Compact". Cancel Auto-Compact for test, please.
Auto-Save dialog is "dialog AFTER save completion" unless "save error" dialog, and has "OK" only. Don't reply Cancel to "Auto-Save Error dialog", and reply Retry, please, if "Auto Save Error".

> >  - mail.purge.ask=true
> OK, changed it

This setting is read upon restart only. Do shutdown/restart after change, please.
(In reply to WADA from comment #61)
> (In reply to David Taber from comment #60)
> > > And If dialog for auto-save appeared while composing, CANCEL it.
> > This implies that my draft will be wiped out if TB or the OS crashes while I'm composing.  Right?
> 
> Sorry. It should be "Auto-Compact". Cancel Auto-Compact for test, please.
OK, done.

> Auto-Save dialog is "dialog AFTER save completion" unless "save error"
> dialog, and has "OK" only. Don't reply Cancel to "Auto-Save Error dialog",
> and reply Retry, please, if "Auto Save Error".
> 
> > >  - mail.purge.ask=true
> > OK, changed it
> 
> This setting is read upon restart only. Do shutdown/restart after change,
> please.
Done.

Since this symptom only happens once every few weeks in normal operation, is there a particular test sequence you want me to go through to perterbate the suspect code?  I'm a little nervous about having your add-in active for a month...
To David Taber. Guide for this bug was;
http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-MsgDB-DBOpen-MsgDBHdr-Tracking.html

This addon is used for problem analysis of bug 905576. Original feature is merely (a) "Dump property value of ,msgFolder, msgDatabase, msgDBHdr etc. to Error Console" only. After it, I added (b) Test for bugs or tool for bug test, and I (c) enhanced and added many features for test of bug 905576.
(a) is safe feature, because it's simply do "for(NM in Obj){Dat[Dat.length]=NM+"="+Obj[NM];}Dump Dat data to Error Console;". MsgDB Event Listener/Event logging consists of (c-1) Event Listener, (c-2) Logging. "(c-2) Logging" is merely "Save data in JavaScript Array if requested, and dump data to Error Console when requested". So it's safe. (c-1) is slightly dangerous, because MsgDB Event Listner/Folder Listener of my addon is registered to Tb. And my code still fails to Unregister it(:-).
However, even if something bad happened, "Restart Tb" clears anything bad, because my addon doesn't do anything automatically. All is initiated by button/menu click, and many actions are "View/Dump Tb's object or JavaScript object data".
Dangerous one is (b). Because purpose of such button/menu is Special Test of Tb, some cases intentionally interfers Tb. For example,
  (i) If Tb sets [Gmail]/Trash as trash folder, I click a menu.
  Menu handler of my addon (ii) removes Trash flag of [Gmail]/Trash,
  and (iii) stores Trash flag in other folder.
  As Tb does do (i) by Folder Re-Discovery, I do (ii)/(iii),
  and as I did (ii)/(iii), Tb does do (i) upon Folder Re-Discovery, ...,
  and my fight with Tb continues in order to use "other than [Gmail]/Trash" as trash folder.
  This is why named WinBackMyTrash :-)
  Other menu tries to issue "Empty Trash" carelessly, regardless of folder state.
  Purpose is to see Tb's behavior when something bad happens,
  so "something bad" is ntentionally generated by button/menu click.
However, all is initiated by button/menu click. "What is done upon startup" is only: Define a global object(window.GlobalVariableName), set data in the object, define functions in object.
So, unless you click dangerous menu intentionally, this addon does do　nothing.

First objective is;
  Force you to see what happens on draft mail when Compact occurs on Drafts folder,
  without lengthy explanation on "how to watch Order Received column value etc.".
So, after you do "save draft multiple time", "Compact Drafts", "save draft multiple times", "Print MsgDBHdr log", you can uninstall addon.
.
FYI.
If edited draft mail is HTML mail and has embed image, phenomenon of bug 817245 can occur upon "save as draft" by Compact.
  Edited mail(messageKey=messageOffset=XXX) :       No embed image              HTML with embed image
  By Compact,
   mail at offset=XXX(==mail of messageKey=XXX) is:       .
     there is no mail at offset=XXX                 old version is not deleted  bug 817245
     other mail is placed at offset=XXX             different mail is deleted
       the other mail has image with same part#                                 bug 766495
       the other mail has text  with same part#                                 bug 799450
       the other mail doesn't have part of same part#                           null is embed
This is phenomena I explaned in comment #5, and I already pointed comment #5 again in my comment #45.
FYI.
In Bug 854798, "phenomenon of old version is not deleted in No embed image case of above table" is not referred long the way. However, needless to say, Bug 854798 is applicable to the phenomenon of "old version is not deleted", because bug 817245 and "old version is not deleted" is essentially issue with same condition. 
> Bug 854798  : Compacting Berkeley Mbox file changes messageKey (to new MsgOffset after compact),
> causing dataloss/privacy problems like bug 817245 due to current design problem of MsgKey=MsgOffset
> (for Berkeley Mbox files)

Because "old version is not deleted due to messageKey value change after Compact of local Drafts folder" is already pretty well known issue, and because Bug 854798 is already opened, there is no need to add comments of "problem of 'old version of draft was not deleted' still occurs" or "Me too" to this bug, if your case is "old version is not deleted due to messageKey value change after Compact of local Drafts folder". 
What you should do first in this bug is to clearly state your case is following case, or not.
  "old version is not deleted due to messageKey value change after Compact of local Drafts folder"
And, it's easy and pretty simple job, if "Order Received" column value is watched/checked carefully with mail.purge.ask=true.
HI Wada--
here's the results of the file-change-save-drafts, compact, file-change-save-drafts sequence. I suspect, though, that what you really wanted was for me to snapshot this every time I did a draft-save.

Please let me know if you do want multiple "prints" of the msgDBhdlr log, or if this data is somehow incomplete.

----------------------- 

2014/03/26 15:13:06.182 GMT WinBackMyTrash#1 : Button_1,click#=14,req=21,label=MsgDBHdr Info of Selected Mail,documentURI=chrome://messenger/content/messenger.xul
List mail info of selected mail

Current selection
  serverURI = mailbox://nobody@Local%20Folders, type = none, isGMailServer = undefined
  trashFolderName = undefined, personalNamespace = undefined
  Mbox=Drafts :  URI = mailbox://nobody@Local%20Folders/Drafts, prettiestName = Drafts, msgDatabase_is_Cached_when_checked = true, AllFlagStates = ( Drafts,Mail,SpecialUse ), expungedBytes = 12100, sizeOnDisk = 81093, lastMessageLoaded = 78672, filePath = D:\Mail\Local Folders\Drafts

msgDBHdr info of selected mail-1 = { 
    [messageKey] = 78672
    [messageOffset] = 78672
    [messageSize] = 2421
    [offlineMessageSize] = 0
    [date] = 1395846229000000
    [dateInSeconds] = 1395846229
    [StringProperty] = { pendingRemoval=, storeToken=78672 }
    [flags] = 1
    [flag_Detail] = { FeedMsg = false, IMAPDeleted = false, Ignored = false, MDNReportSent = false, Read = true, Replied = false, Marked = false, Expunged = false, HasRe = false, Elided = false, Offline = false, Watched = false, SenderAuthed = false, Partial = false, Queued = false, Forwarded = false, Priorities = false, New = false, MDNReportNeeded = false, Template = false, Attachment = false, Labels = false, RuntimeOnly = false }
    [messageId] = 5331F54F.3010501@comcast.net
    [mime2DecodedSubject] = test message-10
}

MsgDBHdr : [xpconnect wrapped nsIMsgDBHdr]
  MsgDBHdr / Property
    Charset = ISO-8859-1
    accountKey = account9
    author = Davo <DavoFanmail@comcast.net>
    bccList = 
    ccList = 
    date = 1395846229000000
    dateInSeconds = 1395846229
    flags = 1
    folder = [xpconnect wrapped nsIMsgFolder]
    isFlagged = false
    isKilled = false
    isRead = true
    label = 0
    lineCount = 35
    messageId = 5331F54F.3010501@comcast.net
    messageKey = 78672
    messageOffset = 78672
    messageSize = 2421
    mime2DecodedAuthor = Davo <DavoFanmail@comcast.net>
    mime2DecodedRecipients = junk@junk.com
    mime2DecodedSubject = test message-10
    numReferences = 0
    offlineMessageSize = 0
    priority = 1
    propertyEnumerator = [xpconnect wrapped nsIUTF8StringEnumerator]
    recipients = junk@junk.com
    statusOffset = 33
    subject = test message-10
    threadId = 78672
    threadParent = 4294967295
  MsgDBHdr / Function
    AndFlags = function AndFlags()
    OrFlags = function OrFlags()
    QueryInterface = function QueryInterface()
    getAuthorCollationKey = function getAuthorCollationKey()
    getProperty = function getProperty()
    getRecipientsCollationKey = function getRecipientsCollationKey()
    getStringProperty = function getStringProperty()
    getStringReference = function getStringReference()
    getSubjectCollationKey = function getSubjectCollationKey()
    getUint32Property = function getUint32Property()
    markFlagged = function markFlagged()
    markHasAttachments = function markHasAttachments()
    markRead = function markRead()
    setBCCListArray = function setBCCListArray()
    setCCListArray = function setCCListArray()
    setPriorityString = function setPriorityString()
    setProperty = function setProperty()
    setRecipientsArray = function setRecipientsArray()
    setReferences = function setReferences()
    setStringProperty = function setStringProperty()
    setUint32Property = function setUint32Property()
I don't need data. I merely want you to understand comment #5/comment #45/comment #64, instead of writing comment for "problem still occurs" repeatedly.
- "button-1 -> MsgDBHdr Info of Selected Mail" is alternative of
     "Write down thread column display of all mails in Drafts".
  If multiple mails are selected at Thread pane, this menu prints info. of selected mails.
- "Print Log" has two modes: Without Clear and With Clear
     Without Clear : Print Log to Error Console. Don't clear saved log lines.
     With Clear    : Print Log to Error Console, and clear saved log lines.
Do procedure like next.
(0) Add a dummy mail to Drafts(call mail-X),
    create a mail, set Subject: 1, save, end composition. This is traced mail.
    Write down Subject, Order Received column value etc. of mail-x and traced mail.
    Clear Error Console, Open Text Editor
(1) Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
    Copy at Error Console, Paste at Text Editor, Clear Error Console
(2) Delete mail-x (deleted mail is needed to actually invoke Compact process)
    "Print Log(with Clear)", Copy at Error Console, Paste at Text Editor, Clear Error Console
    Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
      Copy at Error Console, Paste at Text Editor, Clear Error Console
(3) Edit tracked mail(Suject: 1), change to Subject: 12
    Save as draft (subject change is to increment size by one)
    "Print Log(with Clear)", Copy at Error Console, Paste at Text Editor, Clear Error Console
    Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
      Copy at Error Console, Paste at Text Editor, Clear Error Console
    Write down Suject, Order Received etc. of traced mail
(4) change to Subject: 123, Save as draft,
    "Print Log(with Clear)", Copy at Error Console, Paste at Text Editor, Clear Error Console
    Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
      Copy at Error Console, Paste at Text Editor, Clear Error Console
    Write down Subject, Order Received etc. of traced mail
(5) change to Subject: 1234, Save as draft,
    "Print Log(with Clear)", Copy at Error Console, Paste at Text Editor, Clear Error Console
    Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
      Copy at Error Console, Paste at Text Editor, Clear Error Console
    Write down Subject, Order Received etc. of traced mail
(6) Compact of Drafts folder,
    "Print Log(with Clear)", Copy at Error Console, Paste at Text Editor, Clear Error Console
    Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
      Copy at Error Console, Paste at Text Editor, Clear Error Console
    Write down Subject, Order Received etc. of traced mail
(7) change to Subject: 12345, Save as draft,
    "Print Log(with Clear)", Copy at Error Console, Paste at Text Editor, Clear Error Console
    Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
      Copy at Error Console, Paste at Text Editor, Clear Error Console
    Write down Subject, Order Received etc. of traced mail
(8) change to Subject: 123456, Save as draft,
    "Print Log(with Clear)", Copy at Error Console, Paste at Text Editor, Clear Error Console
    Select all mail at thread pane, "button-1 -> MsgDBHdr Info of Selected Mail",
      Copy at Error Console, Paste at Text Editor, Clear Error Console
    Write down Subject, Order Received etc. of traced mail
(9) At Text Editor, view data, with focusing on messageKey/messageOffset.
    messageSize is altered(incremented by 1) at each save, so it can be used as id of draft version.
    In Log data,
      MsgDBHdrAdded   => mail is newly created and shown at thread pane
      MsgDBHdrDeleted => mail is marked as Deleted in file named Drafts,
                         MsgDBHdr(meta data of mail) is removed,
                         mail is removed from thread pane
    "mail is marked as Deleted in file named Drafts" is removed by Compact.
    And messageKey of mails after the "removed by Compact" mail is currently altered.
    "Delete old version of draft mail upon draft save" is based on messageKey when the draft mail
    is edited.
    "Print Log(without Clear)" is to avoid "Paste at Text Editor after each Print Log" in above steps.
    If "Print Log(without Clear)", "Paste at Text Editor" is mandatory at step (8) only.
Above is possible by "merely watching order received column value". MsgDBHdr dump, MsgDBevent log is merely a helper for understanding phenomena and a tool to change "Write down" to "Click mail/menu, then Copy&Paste".
There is no need to paste your data to this bug.

Is your problem same as "old draft is not deleted by above procedure with Compact"? Or absolutely different problem?
Hi,
Thanks for the detailed procedure...it made it much clearer what you were trying to test for.
Of course, TB properly handled all of the draft versions before and after compacting.  I suppose if I repeated the procedure endlessly for several weeks I could cause it to fail.
As far as I'm concerned, you and I have already spent too much time and email traffic in order to discover the absence of a failure.  Since my issue is so intermittent, reproducing it is not worth the effort.
Thanks again for all your effort.
(In reply to David Taber from comment #68)
> Of course, TB properly handled all of the draft versions before and after compacting.

You didn't see "not-deleted previous version of draft" actually with actually same procedure I wrote?
If so, it's funny. That is pretty consistent "Steps to reproduce Bug 854798, and bug 817245, bug 799450, bug 766495"...

Was messageKey/messageOffset value changed after Compact?
Did you make "deleted mail in Drafts before Compact of Drafts" with "auto-compact=Off" or with "mail.purge.ask=true is surely effective"?
If msgFolder.expungedBytes=0, any of "Compact of context menu", "Compact Folders", Auto-Compact, won't invoke Compaction process of the folder. (expungedBytes can be known by button-1, MsgDB Info of Selected Folder)
Was EventType=CompactCompleted logged in MsgDB Event Listener log?
If yes, problem of "Compact was not executed(expungedytes is not reset to ZERO, kept as-is) even though CompactCompleted event is notified" might happen, which I saw several times when I forced contention of folder between "Bulk filter move of many mails" and "Compact Folders", on both "Move Source Folder" and "Move Target Folder".

> Since my issue is so intermittent, reproducing it is not worth the effort.

All of Bug 854798, and bug 817245, bug 799450, bug 766495, are pretty intermittent problem, because problem happens only when Compact happens on relevant folder while composing mail and only when Compact actually changed messageKey of relevant version of mail data. It's simply sure/consistent/pretty simple/pretty easy "[Steps to reproduce problem]" is already well known, because cause of problem is already pretty clearly known.

My purpose is; surely rule out such already well known problem from this bug, in order to avoid useless "me too" comments in this bug.
In order to surely rule out a problem, you have to know what is the "a problem should be ruled out".
Following is sufficient(rather needed) as your daily check for this bug.
  With mail.purge.ask=true is surely effective, if "confirmation dialog before start Auto-Compact"
  is shown, and if you are composing mail when the dialog is shown,
  watch Order Received column value of relevant mail data in relevant folder carefully,
  (current version of draft, forwarded/replied mail, "Edit As New" mail etc.)
  and Reply OK to invoke Auto-Compact.
  If possible, check expungedBytes of relevant folder, messageKey/messageOffset of relevant mail,
  before Reply OK, and after Reply OK.
If problem of "old draft was not deleted due to Compact" didn't surely occur but "old draft remained in Drafts folder" happened, it's evidence that other problem surely happened in your environment.
I want to get your answer of "your problem is never bug 817245".
If problem like "contention between auto-save and draft delete after send", it may be hooked by enhancement of my extetion.
  At Composer Window, add button-7 via ToolBar customize. Currently available feature:
      Check whether I'm recycled window by mail.compose.max_recycled_windows=N, or not
  This is a test for implementing "JavaScript code for Composer Window status check".
  This is currently done in Composer Window. "Check from messenger window" is preferable,
  and preparation for it was done, but "check other window's status" is not implemented yet.
This is a reason why I opened my extention to you.
If "even looking simple HdrAdded/HdrDeleted Event log" is too hard for you, requesting you to check with my extention is meaningless... :-)

Characteristics of Auto-Compact.
- mail.purge_threshhold_mb is "Total MegaBytes size of expungedBytes of all folders in entire Tb".
- Auto-Compact is never invoked for one hour, after schedule of an Auto-Compact process.
- Auto-Compact / Compact Folders / Compact executes compaction on folder of 0<expungedByte
Problem in Drafts.
  draft save = "delete old version" + "create new version". i.e. expungedBytes is always positive.
  So, if Auto-Compact is invoked, compact operation surely happens on Drafts folder,
  unless Drafts folder is "just after compacted" state.
Problem in mail.purge.ask=true : Frequent dialog is annoying
So, following is better for checking of this bug.
  Set appropriate(sufficiently large) value in mail.purge_threshhold_mb
  Upon each restart of Tb at morning, "Compact Folders".
I believe frequency of dialog will be reduced by this, even with mail.purge.ask=true.

As I wrote in bug 931303 comment #56, because of current design/implementation, if POP3 mail download occurs and filter move by "Filter after Junk Classification" occurs, and if Compact is never executed, file size of Inbox continuously grows silently, even when all mails are moved to other folder by message filter and number of mails in Inbox=0 is kept. (I thought filter move of POP3 does truncate of Inbox upon filter move of each mail, but it was "before Classification" only. "after Classification" used "bulk mail move".)
Please don't disable Auto-Compact blindly, even if you saw problem which is relevant to auto-compact.
(In reply to WADA from comment #69)
> (In reply to David Taber from comment #68)
> > Of course, TB properly handled all of the draft versions before and after compacting.
> 
> You didn't see "not-deleted previous version of draft" actually with
> actually same procedure I wrote?
Correct, there were no "non-deleted previous versions of draft" at any point.

> If so, it's funny. That is pretty consistent "Steps to reproduce Bug 854798,
> and bug 817245, bug 799450, bug 766495"...
> 
> Was messageKey/messageOffset value changed after Compact?
Yes, the Compacting process did change the order-received value.

Also, every save of the message changed the order-received value.


> Did you make "deleted mail in Drafts before Compact of Drafts" with
> "auto-compact=Off" or with "mail.purge.ask=true is surely effective"?
To the best of my knowledge, auto-compacting was off and the mail-purge-ask was true.


> If msgFolder.expungedBytes=0, any of "Compact of context menu", "Compact
> Folders", Auto-Compact, won't invoke Compaction process of the folder.
> (expungedBytes can be known by button-1, MsgDB Info of Selected Folder)
> Was EventType=CompactCompleted logged in MsgDB Event Listener log?
> If yes, problem of "Compact was not executed(expungedytes is not reset to
> ZERO, kept as-is) even though CompactCompleted event is notified" might
> happen, which I saw several times when I forced contention of folder between
> "Bulk filter move of many mails" and "Compact Folders", on both "Move Source
> Folder" and "Move Target Folder".
The compaction process did occur, but only when I explicitly invoked it via the context menu on the Drafts folder.

> 
> > Since my issue is so intermittent, reproducing it is not worth the effort.
> 
> All of Bug 854798, and bug 817245, bug 799450, bug 766495, are pretty
> intermittent problem, because problem happens only when Compact happens on
> relevant folder while composing mail and only when Compact actually changed
> messageKey of relevant version of mail data. It's simply
> sure/consistent/pretty simple/pretty easy "[Steps to reproduce problem]" is
> already well known, because cause of problem is already pretty clearly known.
> 
> My purpose is; surely rule out such already well known problem from this
> bug, in order to avoid useless "me too" comments in this bug.
I concur with your purpose.  

> In order to surely rule out a problem, you have to know what is the "a
> problem should be ruled out".
> Following is sufficient(rather needed) as your daily check for this bug.
>   With mail.purge.ask=true is surely effective, if "confirmation dialog
> before start Auto-Compact"
>   is shown, and if you are composing mail when the dialog is shown,
>   watch Order Received column value of relevant mail data in relevant folder
> carefully,
>   (current version of draft, forwarded/replied mail, "Edit As New" mail etc.)
>   and Reply OK to invoke Auto-Compact.
OK, I have the Order Received column visible in my Drafts folder and will monitor it whenever I am editing draft emails.


>   If possible, check expungedBytes of relevant folder,
> messageKey/messageOffset of relevant mail,
>   before Reply OK, and after Reply OK.
> If problem of "old draft was not deleted due to Compact" didn't surely occur
> but "old draft remained in Drafts folder" happened, it's evidence that other
> problem surely happened in your environment.
Your sentence is a nicely-compounded Boolean, but I think I understand.


> I want to get your answer of "your problem is never bug 817245".
I don't think I've seen the phenomenon described in that bug.  But I suspect it would only occur if I had done a message move or message delete *before* sending the reply/forward mail...and I never do that.

Why do I never do that?  Because if I do a message reply/forward, if I were to move or delete the original message before completing the send, my copy of TB will ALWAYS fail to attach an image.  I don't think that symptom is caused by compacting...I can't believe compacting is running that often or that fast.  To make this clear:
  - create a forward-message of an existing message with an embedded image
  - delete the original message
  - within 1 second, hit send on the forward-message
  - (I think) all the time the image-attach for the forward-message will fail for me


> If problem like "contention between auto-save and draft delete after send",
> it may be hooked by enhancement of my extetion.
>   At Composer Window, add button-7 via ToolBar customize. 

????button 7????  I see only button-1, button-i, and button-3 in the customize-toolbar popup window.  


Currently
> available feature:
>       Check whether I'm recycled window by
> mail.compose.max_recycled_windows=N, or not
>   This is a test for implementing "JavaScript code for Composer Window
> status check".
>   This is currently done in Composer Window. "Check from messenger window"
> is preferable,
>   and preparation for it was done, but "check other window's status" is not
> implemented yet.
> This is a reason why I opened my extention to you.
> If "even looking simple HdrAdded/HdrDeleted Event log" is too hard for you,
> requesting you to check with my extention is meaningless... :-)
> 
> Characteristics of Auto-Compact.
> - mail.purge_threshhold_mb is "Total MegaBytes size of expungedBytes of all
> folders in entire Tb".
> - Auto-Compact is never invoked for one hour, after schedule of an
> Auto-Compact process.
> - Auto-Compact / Compact Folders / Compact executes compaction on folder of
> 0<expungedByte
> Problem in Drafts.
>   draft save = "delete old version" + "create new version". i.e.
> expungedBytes is always positive.
>   So, if Auto-Compact is invoked, compact operation surely happens on Drafts
> folder,
>   unless Drafts folder is "just after compacted" state.
> Problem in mail.purge.ask=true : Frequent dialog is annoying
> So, following is better for checking of this bug.
>   Set appropriate(sufficiently large) value in mail.purge_threshhold_mb
>   Upon each restart of Tb at morning, "Compact Folders".
> I believe frequency of dialog will be reduced by this, even with
> mail.purge.ask=true.
> 
> As I wrote in bug 931303 comment #56, because of current
> design/implementation, if POP3 mail download occurs and filter move by
> "Filter after Junk Classification" occurs, and if Compact is never executed,
> file size of Inbox continuously grows silently, even when all mails are
> moved to other folder by message filter and number of mails in Inbox=0 is
> kept. (I thought filter move of POP3 does truncate of Inbox upon filter move
> of each mail, but it was "before Classification" only. "after
> Classification" used "bulk mail move".)
I have not seen this symptom at all, although I have many filters on all inbound mailboxes.


> Please don't disable Auto-Compact blindly, even if you saw problem which is
> relevant to auto-compact.
I have auto-compacting on, and I expect it to ask me each time (mail.purge.ask=true).

Before I start a compacting cycle, I will look at the Drafts folder and observe changes.  Will report back if I find anything interesting/unexpected.
(In reply to David Taber from comment #70)
> ????button 7????  I see only button-1, button-i, and button-3 in the customize-toolbar popup window.  

button 1 to 3 is for messenger window, and button 7 to 9 is for composer window. Do ToolBar customize at Composer winddow, please.
For data when problem occurred again.
Because Compact is absolutely irrelevant to your case, and local Drafts folder of BerkleyStore, and number of composed mails at same time is not so many usually, all data except "timestamp of delete of old version" is held in file named Drafts.
  From - Sat Mar 01 19:12:49 2014 <= timestamp when this mail data is added to this folder
  X-Mozilla-Status: 0000(before delete) or 0008 or 0009(after delete)
  All mail data follows.
Because you are who composed the mail(s), you can ditinguish mail data even if you composed multiple mails at same time. And, it sounds that "not deleted old draft" is always observed around your mail composition. (it's never "draft disappeared after send, but old version was shown after Repair Folder later) So, "backup of Drafts file/Drafts.msf when problem happened" is useful and pretty important data.

David Taber(bug opener), keep backup of Drafts/Drafts.msf when your problem occurred again, please.

If problem like following,
   Final draft save before send. Offset=XX
   Send, (final draft for send==draft at Offser=XX)         Auto-Save is invoked at same time
      after send completion, delete data at Offset=XX         Save to Offset=YY(>XX).
      save sent mail copy to Sent.                            Because draft at Offset=XX is deleted
                                                              does do nothing.
timestamp of (a) save at Offset=XX, (b) Send request, (c) save to Offset=YY, (d) save to Sent, is pretty important. (a)/(c)/(d) can be obtained from backup of Drafts/Sent, but timestam of "Send request" "send completion" can't be obtainedby ordinal way(at least SMTP:5 log is needed).
Please write down time of your send request, if you feel problem will be reproduced by this Send.
By the way, this case is not "old draft was not deleted". Auto-Save generated exess draft. It's not Obsolete draft. Therefore, this case is different problem from this bug :-)
OK Wada, I will keep track of this and let you know when I find something.
Some clues that I don't think I ever documented here (forgive me if I'm repeating obvious stuff).
* This symptom either doesn't occur at all or occurs with large numbers of draft mails clustered together.
* The clusters will be 5 versions of one draft mail, 3 versions of another, 1 of another, and 17 of another that I kept the edit-window open for a very long time.  Each of the versions will be an auto-save snapshot that should have been wiped out by its successor.
* The symptom will be invisible to me because (1) I don't look in the drafts folder very often and (2) the folder will not show any "new" messages in the folder-browser pain of TB.
* When I open the Draft folder, there isn't an MSF file for it...and when TB creates one all of a sudden the Draft folder will show a whole bunch of "new" (unread) mails
* Elsewhere, I have filed a bug about MSF files disappearing on my system, which they do on a ?monthly? basis (about the same frequency as this bug).  There may be a connection, there may not.
(In reply to David Taber from comment #73)

It sounds for me your case is Comment #19, by out-dated-msf condition.

(1) bug 261419 / bug 495760
    If "outdated msf condition" exists, "Flter move" silently appends mail data to msgStore file,
    without invoking Rebuild-Index, without writing MsgDBHdr of added mail to .msf file.
    Filter move(after Classification), Manual filter run, Manual mail Copy/Move, 
    Save sent mail copy to Sent, uses DoCoy(bulk mail copy/move).
    Note: "Filter move(before Classification)" doesn't use DoCopy, and bug 931303 occurs instead.

(2) Characteristics of "Draft Save".
    - "draft save" also uses DoCopy. ("add new version" part = same as "Save sent mail copy")
    - Because of Special Folder, Unified Folder is opene at same time.
    - HdrAdded event is notified to Event Listener upon each draft save.
    - HdrDeleted event is notified to Event Listener when each delete of old version.
    - After mail composition window close, Drafts folder is quickly closed.
    "Difference from Sent Folder in bug 905576" is following only:
    - HdrDeleted event is notified to Event Listener when each delete of old version.
> MSGDB:5,MsCopyervice:5 log,
> and DBOpened/HdrAdded/HdrDeleted event log by MsgDB Event Listener of my addon.
> NSPR log was logged by DebugView, so timestamp=loacl time(GMT+09:00).
> (1) Start mail compsition Save #1
>            00000000 16:16:10.262 [1664] 0[e0f140]: request 63037c0 DoCopy - src  dest mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts numItems 0 type=1 
>            00000001 16:16:10.262 [1664] 0[e0f140]: nsMsgDatabase::Open(C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\wkeci8t7.ZZZ\Mail\pop.ops.dti.ne.jp\Drafts.msf, FALSE, 76d62f0, TRUE) 
>            00000004 16:16:10.340 [1664] 0[e0f140]: nsMsgDatabase::Open(C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\wkeci8t7.ZZZ\Mail\smart mailboxes\Drafts.msf, FALSE, 76d6420, TRUE) 
> DateTime=2014/03/29 07:16:10.298 GMT, EventType=DBOpened, MsgDB=mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts
>            00000008 16:16:10.340 [1664] 0[e0f140]: NotifyCompletion - src  dest mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts 
>            00000009 16:16:10.340 [1664] 0[e0f140]: request 63037c0 Clearing OK request - src  dest mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts numItems 0 type=1 
> (2) Save #2
> DateTime=2014/03/29 07:16:10.347 GMT, EventType=MsgDBHdrAdded, messageKey=13073, messageOffset=13073, messageSize=748, summaryValid=false, MsgDB=mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts
>            00000010 16:16:19.012 [1664] 0[e0f140]: request 6303e00 DoCopy - src  dest mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts numItems 0 type=1 
> DateTime=2014/03/29 07:16:19.026 GMT, EventType=MsgDBHdrAdded, messageKey=13821, messageOffset=13821, messageSize=749, summaryValid=false, MsgDB=mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts
>            00000011 16:16:19.028 [1664] 0[e0f140]: NotifyCompletion - src  dest mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts 
>            00000012 16:16:19.028 [1664] 0[e0f140]: request 6303e00 Clearing OK request - src  dest mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts numItems 0 type=1 
> DateTime=2014/03/29 07:16:19.078 GMT, EventType=MsgDBHdrDeleted, messageKey=13073, messageOffset=13073, messageSize=748, MsgDB=mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts
> DateTime=2014/03/29 07:16:19.083 GMT, EventType=DeleteOrMoveMsgCompleted, FolderURI=mailbox://soarex%40ops.dti.ne.jp@pop.ops.dti.ne.jp/Drafts
> (3) Close composer window
>            00000013 16:17:24.918 [1664] 0[e0f140]: closing database    C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\wkeci8t7.ZZZ\Mail\pop.ops.dti.ne.jp\Drafts.msf 

(3) It looks that bug 905576 can occur on any local mail folder.
    We have log data of;
    (i) Merely     "one mail move by message filter"    (N cases)
                or "two mail move by message filter"    (1 case)
                or "save of one sent mail copt to Sent" (2 case)
           "filter move" : both "after Classification" case and "before Classification" vase
                           are perhaps comtained in log.
    (ii) After it, 
      type-A : folder is closed by "filter move" quickly after end of adding mail,
               then the folder is immediately opened by someone, and is closed after a while.
      type-B : "closing database" is not logged by MSGDB:5 logging after end of adding mail,
               then the folder is kept open, and is closed after a while.  
    (iii) After it, 
       0 to N "open log, then immediate 'closing database' log" by MSGDB:5 logging
    (iv) Finally, when folder is opened by Virtual Folder open, Rebuild-Index is executed.
    That's all, in any case of that bug!

(In reply to David Taber from comment #51)
> For me: win7pro 64 bit with indexing on, POP, glodia on, and the drafts folder in encrypted directory.

If encrypted file, I think "decription of entire file data is executed upon file open" and "encryption of entire file data is executed upon file close".
Right?

Bug 905576 is phenomenon in SMB1 file share environment.
Drafts is special folder, so local Drafts folder is always a search target folder of "Unified Drafts Folder".
If my assumption is right, and if "required time for file open/close" is relevant to Bug 905576, Bug 905576 may happen on your Local Drafts folder.
Because Auto-Compact doesn't occur so frequently in your environment, if "outdated msf condition" is produced on Drafts folder, Rebuild-Index of Drafts will usually not be invoked until you explicitly open Drafts folder.
So, once "outdated msf condition" is generated in Drafts folder, I think probability of "draft save occurs when outdated msf condition exits in Drafts" is relatively high in your environment.
Hi Wada,
Thanks for all the analysis of this behavior. I agree with your speculations & conclusions.
One thing I forgot to mention:  there are no filters on the DRAFTS folder, therefore the filter-related bugs should be irrelevant...right?
I have turned file-system encryption OFF on the DRAFTS folder, so we can avoid any issues related to that as well.
(I do suspect that windows' encryption would have to create a new file on every edit [I don't know if it's smart enough to change just some of the contents of a file without create new one, destroy existing one.])
(In reply to David Taber from comment #75)
> I have turned file-system encryption OFF on the DRAFTS folder, 
> so we can avoid any issues related to that as well.

If suspect is killed, we can't find culprit...
Keep encryption ON, and do following, please.
(0) Show dialog after "save draft", keep auto-save=on
(1) After restart of Tb, View/Folder\Unified, Ecpand Unified Drafts folder, click Drafts, shift+click Templated(all draft folders are selected at folder pane), button-1, Add Event Listener => DBOpened, HdrAdded/HdrDelete event on all draft folders is logged.
(2) Do new mail composition without explicit Draft folder open, unless you need to Edit existent draft mail. While composing mail, when "dialog after save" is shown, check MsgDB event log sometimes, when you feel something bad will happen.
If "outdated msf condition" exists, MsgDatabase open fails. Because "draft save" silently appends draft data to file named Drafts without updating Drafts.msf(unable to update because of "open failure"), and because DBOpened/HdrAdded/HdrDeleted event is not notified to event listener when MsDatabase is not opened, funny phenomenon occurs.
  Draft Save/Auto-Save says "save normally ended",
  but no log for DBOpened, HdrAdded event on Drafts folder.
If Drafts is opened at this timing, Rebuild-Index is invoked.
So, when such phenomenon is observed, we can get problem analysis data merely by;
 Keep backup of Drafts/Drafts.msf, open Drafts, keep backup of Drafts/Drafts.msf after rebuild-index

In Bug 905576, "problem occurs on which folder" is unpredictable, so data gathering was pretty tough work. Your case is Draft only issue. So, it's needed to track Drafts only.
I couldn't imagine reason why "outdated msf condition of Drafts" is relatively easily produced in your environment. But after analysis of Bug 905576, I can imagine following scenario.
      Open Drafts
      Filal draft save before send(messageKey=F)
 A      => HdrAdded is notified
 |    Send             |                            Auto-save starts at same time
 |l                    V                            new draft is saved(messageKey=G>F) 
 |o           someone accesses draft mail             => HdrAdded is notified                 A
 |n    Delete final draft(messageKey=F)                           |                           |
 |g     => HdrDeleted is notified                                 V                           |short
 |                                                             someone accesses draft mail    |
 V    Close Draft quickly  <-- can occur at almost same time ->                               V
                                                    try to delete previous version(messageKey=F)
      Save sent mail copy to Sent
      Close composition window

(In reply to David Taber from comment #73)
> * When I open the Draft folder, there isn't an MSF file for it...

When Drafts.msf file is deleted, if "Explicit folder open by folder click" is first opener of Drafts, Rebuild-Index is invoked automatically. However, if "draft save" is first opener, "deleted .msf" state == "no mail in Drafts" for "draft save". So, status of "offset=0 to current Draft file size"=merely garbage data, and status of "number of mail in Draft=1", is generated. And, IIRC, "outdated msf condition" is cleared by "draft save". i.e. if compact happens, all old draft mail is lost by Compact.

Did you see "there isn't an MSF file for it..." always when you saw rebuild-index of Draft folder?
(In reply to WADA from comment #76)
> (In reply to David Taber from comment #75)
> > I have turned file-system encryption OFF on the DRAFTS folder, 
> > so we can avoid any issues related to that as well.
> 
> If suspect is killed, we can't find culprit...
> Keep encryption ON, 
DONE


and do following, please.
> (0) Show dialog after "save draft", keep auto-save=on
> (1) After restart of Tb, View/Folder\Unified, Ecpand Unified Drafts folder,
> click Drafts, shift+click Templated(all draft folders are selected at folder
> pane), button-1, Add Event Listener => DBOpened, HdrAdded/HdrDelete event on
> all draft folders is logged.
I don't have a "templated" folder, although i do have "templates"  using that one for the Event Listener in addtion to "draft".  On button-1, I don't have an item called "Add Event Listener", and there is no way to spec "DBOpened, HdrAdded/HdrDelete" that I can see.  The closest thing I can see in Button-1 menu items is "MsgDB:  Add Listener" but there's nothing more to specify on that item

Consequently, I am not proceding with the rest of what you write below (as it is unlikely to yeild any results yet).

> (2) Do new mail composition without explicit Draft folder open, unless you
> need to Edit existent draft mail. While composing mail, when "dialog after
> save" is shown, check MsgDB event log sometimes, when you feel something bad
> will happen.
> If "outdated msf condition" exists, MsgDatabase open fails. Because "draft
> save" silently appends draft data to file named Drafts without updating
> Drafts.msf(unable to update because of "open failure"), and because
> DBOpened/HdrAdded/HdrDeleted event is not notified to event listener when
> MsDatabase is not opened, funny phenomenon occurs.
>   Draft Save/Auto-Save says "save normally ended",
>   but no log for DBOpened, HdrAdded event on Drafts folder.
> If Drafts is opened at this timing, Rebuild-Index is invoked.
> So, when such phenomenon is observed, we can get problem analysis data
> merely by;
>  Keep backup of Drafts/Drafts.msf, open Drafts, keep backup of
> Drafts/Drafts.msf after rebuild-index
> 
> In Bug 905576, "problem occurs on which folder" is unpredictable, so data
> gathering was pretty tough work. Your case is Draft only issue. So, it's
> needed to track Drafts only.
> I couldn't imagine reason why "outdated msf condition of Drafts" is
> relatively easily produced in your environment. But after analysis of Bug
> 905576, I can imagine following scenario.
>       Open Drafts
>       Filal draft save before send(messageKey=F)
>  A      => HdrAdded is notified
>  |    Send             |                            Auto-save starts at same
> time
>  |l                    V                            new draft is
> saved(messageKey=G>F) 
>  |o           someone accesses draft mail             => HdrAdded is
> notified                 A
>  |n    Delete final draft(messageKey=F)                           |         
> |
>  |g     => HdrDeleted is notified                                 V         
> |short
>  |                                                             someone
> accesses draft mail    |
>  V    Close Draft quickly  <-- can occur at almost same time ->             
> V
>                                                     try to delete previous
> version(messageKey=F)
>       Save sent mail copy to Sent
>       Close composition window
> 
> (In reply to David Taber from comment #73)
> > * When I open the Draft folder, there isn't an MSF file for it...
> 
> When Drafts.msf file is deleted, if "Explicit folder open by folder click"
> is first opener of Drafts, Rebuild-Index is invoked automatically. However,
> if "draft save" is first opener, "deleted .msf" state == "no mail in Drafts"
> for "draft save". So, status of "offset=0 to current Draft file size"=merely
> garbage data, and status of "number of mail in Draft=1", is generated. And,
> IIRC, "outdated msf condition" is cleared by "draft save". i.e. if compact
> happens, all old draft mail is lost by Compact.
> 
> Did you see "there isn't an MSF file for it..." always when you saw
> rebuild-index of Draft folder?
(In reply to David Taber from comment #77)

Sorry, it was "MsgDB: Add Listener". Read guide pointed in comment #63, please.
> http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-MsgDB-DBOpen-MsgDBHdr-Tracking.html

> I don't have a "templated" folder

That's procedure to select all actual Drafts folders, without opening actual Drafts folders, by a few mouse clicks.
Corrected procedure.
   1. Expand Unified Drafts
   2. Click next Unified Folder to Drafts (Unified Folder under last actual Drafts folder)
      => This Unified Folder is shown at Thread pane == all assocated actual folders are opened
   3. Shift+Click Unified Drafts => Unified Drafts nor any actual Drafts is not opened.
Because MsgDatabase is kept open once opened until folder is closed by idle threshold, MsgDatabase open by operation for teting is better avoided. If you don't touch Drafts, you can't be culprit :-)

Step (2) is for "keep Drafts folder 'closed state' as long as possible", "don't interfere draft save job by your manual open of Drafts", "keep state of 'who acccess Drafts' is 'draft save' only".
If Drafts is touched by 'draft save' only and you never touched, and if Drafts is broken, culplit is surely 'draft save' instead of you :-)

Because Drafts is always opened before draft save if Drafts is not opened yet, I don't think "Drafts open by you" won't affect on problem.
However, if Drafts is opened and content is shown at Thread pane, "Close after draft save" may not be invoked.
Because problem looks to occur due to "quick close after draft save" and "open request invoked by message access just after HdrAdded event notification", "open/close by other than 'draft save'" is better avoidd.
(In reply to WADA from comment #78)
> (In reply to David Taber from comment #77)
> 
> Sorry, it was "MsgDB: Add Listener". Read guide pointed in comment #63,
> please.
> > http://www.h2.dion.ne.jp/~radon/mozilla/tb-addon/WinBackMyTrash-0-1-0-MsgDB-DBOpen-MsgDBHdr-Tracking.html
> 
> > I don't have a "templated" folder
> 
> That's procedure to select all actual Drafts folders, without opening actual
> Drafts folders, by a few mouse clicks.
> Corrected procedure.
>    1. Expand Unified Drafts
Sorry to be stupid here, but there's nothing to expand.  The folder tree I have is
Local Folders>Drafts

I have changed the View>Folder options to every one that is listed and there is never a parent Drafts folder, nor any children.

IAC, I have simply added a listener to the Drafts folder that I have.


>    2. Click next Unified Folder to Drafts (Unified Folder under last actual
> Drafts folder)
>       => This Unified Folder is shown at Thread pane == all assocated actual
> folders are opened
>    3. Shift+Click Unified Drafts => Unified Drafts nor any actual Drafts is
> not opened.
> Because MsgDatabase is kept open once opened until folder is closed by idle
> threshold, MsgDatabase open by operation for teting is better avoided. If
> you don't touch Drafts, you can't be culprit :-)

Will keep Drafts folder closed, but will monitor for missing Drafts.msf file.
You look to have Drafts in Local Folders only. If so, there is no need to use Unified Folder in order to select multiple Drafts only by a few clicks.

> will monitor for missing Drafts.msf file.

I recommend you to see event log when Drafts.msf is deleted before you start daily check.
1. button-1, Toggle Alert => when summaryValid=false is detected upon DBOpened event, show alert.
2. Delete Drafts.msf when Drafts folder is not opened yet
   button-1 "Show Event Status" is usable to know opened or not
3. Click Drafts at folder pane => Rebuild-Index is invoked
4. When Rebuild-Index is executed, msgDatabase.summaryValid=false is temporarily set.
   So, summary=false is detected by my addon upon DBOpened event is notified
   => alert is shown one or a few times.
5. Repair Folder of Drafts => same alert is shown
   If Repair Folder, AnnouncerGoingAway event is notified to listener.
   Because msgFolder nor msgDatabase is not passed to event listener, we can not know where the event
   happened, but this kind of event is an evidence of "something special" happened.
   Notification of this event is seen by Rename folder, Delete folder etc.
5. View event log
Alert is annoying. Disable Alert by "Toggle Alert" after test, please. Log for last "detection of summary=false" is kept, and is shown by "Show Event Status". I don't think annoying Alert is needed in your daily check.
Hi, an update.  Not sure whether there is much information value here...but it is data.

In my daily check, I looked in the windows file system for the Drafts and Drafts.msf files after they hadn't been viewed in the UI or compacted for several days.  
The Drafts file was appropriately large (54k) but the Drafts.msf file was empty (not gone, but 0kB)!  I then hit the compact button and Drafts shrank appropriately and the Drafts.msf file grew to a reasonable size (3k).  No funny stuff in the UI either before or after the compacting.

I then go look in the Drafts.mozmessages folder, and find maybe 10 old files sitting around.  Some of them 3 years old, so not legitimate drafts.  These could have occurred because of TB upgrades, even system migrations (really, been through 3 PCs in that time)...I delete the old junk, under the idea that these files may have been confusing some part of TB.  (Note the files were all fine, not themselves corrupted, but their zombie existence could cause random complications within TB.)
This exact same thing just started happening to me (multiple
copies of incremental Drafts are left behind after I send).

Worked fine in Thunderbird 31.0.  Happens very often with 
Thunderbird 31.1.2.  I'm on Mac OS X 10.6.8 (Snow Leopard).

This ticket looks very old (2009) so maybe I should be creating
a new ticket since this seems like a recent regression.  But 
since my problem exactly matches the original problem description
and since the ticket is not yet closed, I'm adding it here.

If it doesn't get any attention, I'll create a new ticket.

Feel feel free to contact me for more info, or to have me do 
experiments.  Thanks for all your hard work!  Thunderbird is 
far and way the best email I client I've ever seen in 30+ years
of email usage.

--Fred Stluka
fred@bristle.com
Yes, have noticed this symptom flaring up again.  I'm on Win7 pro 64 bit.

Fred, are you storing your mail files (particularly the "local folder" stuff where drafts live) in any special area of the Mac file system?  Does that folder have any special attributes turned on (e.g., compression, encryption)?
David,

No, nothing special.  No compression.  No encryption.  No
special permissions or ACLs.

Also, the Drafts mail folder is not in "Local Folders".  It
is in the directory tree of this mail account.  I have 
Thunderbird configured to pull mail from several accounts 
(Gmail, Comcast, etc.), each into its own tree.   I don't
use "Local Folders" for anything.

The only thing I did that might be unusual is that I edited
the profiles.ini file to refer to a nonstandard location for 
my mail files.

On the Mac, all of my files are in the /Users/fred tree, but
I create a subtree of that called /Users/fred/fred where I keep
all important files that I back up more often than the rest.

So the install of Thunderbird created a folder in the standard 
place:
    /Users/fred/Library/Thunderbird

that contains a file called profiles.ini.

I edited that file to look like:

[General]
StartWithLastProfile=1

[Profile0]
Name=default
IsRelative=0
Path=/Users/fred/fred/Mozilla/TBird/Profile

and moved the folder:
    /Users/fred/Library/Thunderbird/Profiles/9e0fvy36.default
to
    /Users/fred/fred/Mozilla/TBird/Profile

Has been that way for years, with never a problem.

Anything else I can answer for you?
--Fred
(In reply to Fred Stluka from comment #84)
> Also, the Drafts mail folder is not in "Local Folders".
> It is in the directory tree of this mail account.
> Profile directory = /Users/fred/fred/Mozilla/TBird/Profile

As clearly stated in bug summary of this bug, this bug is for "Drafts folder which is local mail folder in Tb", is never for "IMAP Drafts folder".
(Q1) Where is your "the directory tree of this mail account" located? Local HDD of you PC? Or File Share server?

/Users/<usernme> is usually local HDD directory, so I guess answer is "on local HDD", regardless of "internal HDD device" or "external HDD device".

FYI.
If Drafts/Drafts.msf is located on File Share server, bug 905576 can occur, which produces "outdated msf condition".
If "outdated msf condition" happens at a local mail folder, as I already wrote in comment #74, bug 261419 / bug 495760 can occur.
If "outdated msf condition" happens at a local mail folder, and if it's detected by message filter(Filter before Junk Classificaation), as I already wrote in comment #74, bug 931303 can happen.

(Q2) No file access by other than Thunderbird toTb's Drafts/Drafts.msf file for relevant account in Tb?
   other than Thunderbird : Virus scan by AV  software, Auto-backup software etc.
   They can open Tb's Drafts.msf file/Drafts file just before Tb tries to open Drafts.msf file/Drafts file.
   If Tb's file open of Drafts.msf file is interfered by other software, "outdated msf condition" is internally generated.
WADA,

> As clearly stated in bug summary of this bug, this bug is 
> for "Drafts folder which is local mail folder in Tb", is 
> never for "IMAP Drafts folder".

Does that mean I'm experiencing an unrelated bug and should
open a new ticket?


> (Q1) Where is your "the directory tree of this mail account" 
> located? Local HDD of you PC? Or File Share server?

In this case, it is:
    /Users/fred/fred/Mozilla/TBird/Profile/mail/mail.bristle.com/Drafts
which is a local file on the Mac (not Windows PC) HDD.

> If "outdated msf condition" happens at a local mail folder, 
> as I already wrote in comment #74, bug 261419 / bug 495760 
> can occur.
> If "outdated msf condition" happens at a local mail folder, 
> and if it's detected by message filter(Filter before Junk 
> Classificaation), as I already wrote in comment #74, bug 
> 931303 can happen.

I do have a lot of mail filters, but none of them operate on
the Drafts folder, either reading from it, or moving/copying 
messages there.  I do not have any Junk classification enabled.
So no messages are ever moved to a Junk folder.

> (Q2) No file access by other than Thunderbird toTb's 
> Drafts/Drafts.msf file for relevant account in Tb?
> other than Thunderbird : Virus scan by AV  software, 
> Auto-backup software etc.
> They can open Tb's Drafts.msf file/Drafts file just before 
> Tb tries to open Drafts.msf file/Drafts file.
> If Tb's file open of Drafts.msf file is interfered by other 
> software, "outdated msf condition" is internally generated.

No AV.  Yes, there are backups happening, but they should be
readonly opens of the files.  I'm using Mac's "Time Machine"
software, writing via WiFi to an external "Time Capsule" hard 
drive.

Is it possible the backups are relevant when they only read
the files, never write to them?

Thanks!
--Fred
(In reply to Fred Stluka from comment #86
> > As clearly stated in bug summary of this bug, this bug is 
> > for "Drafts folder which is local mail folder in Tb", is 
> > never for "IMAP Drafts folder".
> 
> Does that mean I'm experiencing an unrelated bug and should open a new ticket?

Yes, if your problem is on IMAP Drafts folder about "old draft mail which was not deleted".

IMAP folder case is different from local mail folder case.
   type=pop/none :
        Unieque identifier of mail = messageKey assigned by Tb, but mail data is accessed using messageOffset.
              Both messageKey(Drafts.msf file) and messageOffset(Drafts file) are relevant to "delete old draft mail".
              Both "remove MsgDBHdr for the messageKey from Drafts.msf" and 
              "Setting EXPUNGE bit=On in X-Mozilla-Status header in mail data" is mandatory to delete old draft mail.
   type=imap:
        Unieque identifier of mail = messageKey = UID of the mail at IMAP server which managed by server and is never alterd.
              "delete old draft" is done by "uid store UIDOfOldDraftMail +Flags(\Deleted)" command o IMAP.
              Once the IMAP command is issued, all is up to IMAP server, 
              because \Deleted flag is set at IMAP server and it's sync'ed again by Tb sooner or later.
        However, there are some known/unknown problems in getting/setting the UIDOfOldDraftMail value in IMAP,
        and there are some IMAP unique issues/problems in re-sync with server after delete old draft mail.
Some IMAP unique issues/problems are already reported to other bugs, so this bug is kept only for "old draft mail is not deleted in type=pop3/none".

 
> Yes, there are backups happening, but they should be readonly opens of the files.

"Read only or not" is usually absolutely irrelevant to "interfere of Tb's write open of XXX and XXX.msf file", because any read open prohibits "Write open by Tb's process while file is opened in read mode by someone" unless "Shared Mode which is used by Tb" is properly used by other software, and because Tb's "return code check after OS API call" is surely insufficient.
WADA,

>> Does that mean I'm experiencing an unrelated bug and should open a 
>> new ticket?
> Yes, if your problem is on IMAP Drafts folder about "old draft mail 
> which was not deleted".
> ...
> Some IMAP unique issues/problems are already reported to other bugs, 
> so this bug is kept only for "old draft mail is not deleted in 
> type=pop3/none".

I'm using POP3, not IMAP.  Sounds like I should NOT open a new ticket.
True?

> "Read only or not" is usually absolutely irrelevant to "interfere 
> of Tb's write open of XXX and XXX.msf file", because any read open 
> prohibits "Write open by Tb's process while file is opened in read 
> mode by someone" unless "Shared Mode which is used by Tb" is 
> properly used by other software, and because Tb's "return code 
> check after OS API call" is surely insufficient.

Thanks for the detailed explanation!

I would expect that the standard Apple "Time Machine" software that
runs automatically without being triggered by the user MUST use 
Shared Mode.  Otherwise, it would interfere with lots of other 
running processes and I'd be getting errors from them or seeing 
data problems with them.  It runs automatically once an hour or
so.  This problem seems to affect only the Drafts folder of 
Thunderbird, not any of my other Thunderbird folders, including my 
inboxes and various folders that I filter incoming messages to
Also, doesn't seem to affect any program except Thunderbird.

Conflicting with backups does not seem like a likely explanation.

I'll pay attention to whether it happens when I have Time Machine
turned off.

Anything else you want me to try, to help debug it?

Thanks!
--Fred
(In reply to Fred Stluka from comment #88
> I'm using POP3, not IMAP.  Sounds like I should NOT open a new ticket.
> True?

Yes, you are right.
This bug was kept open, and was limited to non-IMAP cases, in order to analyze problems what you saw.
    old draft mail was somehow not deleted, even though local Drafts folder.
If normal/ordinal environment/situation, phenomenon of "old draft mail in local Drafts was not deleted" should occur only when;
   1. A draft mail(call draft-mail-X) is saved at Offset=A.
   2. Edit the draft-mail-X at Offset=A.
   3. While editing the draft mail, Compaact of local Drafts folder happens. (this is pretty ordinal if Auto-Compact is enabled)
   4. By "Compaact of  local Drafts folder," Offset of the draft-mail-X is changed to Offset=B(less than A).
   5. When new version of draft mail is saved,
   5-a．If "mail data at Offset=A after Compact of Drafts" is not vald mail data,
           "delete old draft mail at Offset=A" fails, because valid "mail data at Offset=A" doesn't exist any more.
   5-b. If "mail data at Offset=A after Compact of Drafts" is "valid mail data of other draft-mail-Y",
           draft-mail-Y(at Offset=A) is deleted, and draft-mail-X(at Offset=B<A) is not deleted.
Note:
This is known phenomenon/issue/problem in current implementation, and can be called "current restriction in Thunderbird" because design change/implementation change is pretty hard.
WADA,

I have always had Thunderbird configured to prompt me before
compacting, and I pay close attention to only allowing it when
nothing else seems to be going on, like fetching new email.
But, I'll make a point of not allowing it to compact while I'm 
still composing an outgoing message, and and will see if that 
helps.

Does your scenario explain why I would get 10 or more copies
in Drafts, when it takes me 15 minutes to compose an email 
message?  Each copy is slightly more complete than the previous
one, as I gradually complete typing the mail message.  I can 
understand how the compacting process would cause the current 
draft to not be deleted because it moved it to a new offset, 
but unless compact occurred 10 times in the 15 minutes (which
it did not), how would I end up with 10 undeleted drafts of the 
message?

Thanks!
--Fred
Hi Fred,
The root cause of the multiple versions is auto-saving of drafts, which seems to occur after a few minutes of an idle composition window (at least on Windows).  That auto-saving behavior has been around a loooooooong time.

I believe that somehow the T'bird cleanup routine that runs when you actually hit "send" is not able to remove the intermediate versions of the saved file, either because of an internal T'bird issue or (more likely) some sort of file system locking that is preventing deletion of the files.  When compacting occurs, T'bird recognizes that the MSF structure doesn't "see" those files, so it rebuilds the MSF to show the bogus intermediary litter.

Now, T'bird should be able to "ride through" this issue and do the right thing...but it doesn't.  That's what I think Wada is trying to troubleshoot.
David,

Thanks for the info!  I may be wrong, but I always assumed that 
the behavior of Thunderbird was to delete each previous draft
of a message when it saved the next one, and then to finally 
delete the last draft when it did the send.  It sounds like you 
are saying that it usually keeps all of the drafts, and then 
deletes them all when it sends.  True?  Or did I minunderstand?

In any case, I'll keep an eye on my Drafts folder and see which
seems to be happening for me.  

Thanks!
--Fred
Fred,

Well, in truth all of my statements are pure conjecture.  But I've been watching this behavior in T'bird since I first filed this bug 5.5 years ago, and I don't know any other way this symptom could occur if T'bird were successfully overwriting its temp-save files.  As WADA indicates, at least part of the problem comes from a bad-file-pointer issue.

I'd be interested, in the abstract, to know what the code actually does.  But it hasn't been worth my time to read what the code actually says...
(In reply to Fred Stluka from comment #90)
> Does your scenario explain why I would get 10 or more copies in Drafts,
> when it takes me 15 minutes to compose an email message?

If problem due to "Auto-Compact of Drafts folder while editing draft", phenomenon should be "one excess draft mail only per a draft mail editing", because next Auto-Compact is invoked after one hour from previous Auto-Compact execution(Auto-Compact waits for at least one hour after an auto-compact execution).
If cause is contention between "new draft save/old draft delete by Auto-Save" and "old draft delete after mail send", problem should occur upon mail sending.
So, if "10 or more copies" is generated in local Drafts folder after start of mail composition(or draft editing), it may be:
  (1) Outdated msf condition exists in Drafts folder, or Tb's write open of Drafts.msf is interfered by other software or Tb himself.
  (2) Draft save occurs, and data of new draft mail is appended to Drafts file, but Drafts.msf is not updated due to (1).
        => outdated msf condition continues.
  (3) New draft save occurs, and data of the new draft mail is appended to Drafts file, but Drafts.msf is not updated due to (1).
        Tb tries to delete previous version of draft mail, but "set Expunge bit=On in X-Mozilla-Status:" fails due to (1),
        and Drafts.msf is not updated due to (1).
        => outdated msf condition continues.
  (4) (3) repeatedly occurs if Auto-Save is enabled.
  (5) Drafts folder is explicitly opened by folder click of Drafts folder at Folder Pane.
        => Because outdated msf condition, Internal Rebuild-Index(==Repair Folder) is invoked.
              Because "Expunge bit=On" is not set in X-Mozilla-Status: header of old draft mails,
              they are shown as valid draft mails in Drafts folder.

When, How, did you find "10 or more copies"?
Was status of Drafts folder normal and actually healthy(no outdated msf condition, Size of Drafts.msf is not ZERO) before you started mail composition or draft mail editing?
No interfere on Tb's Drafts.msf file open by other software?
(You can easily check it by monitoring tool for file access. For example, Process Monitor if MS Win, opensnoop if Mac OS X)
(In reply to Fred Stluka from comment #92
> Or did I minunderstand?

You didn't misunderstand.
   Draft save is;
      1.  Start mail composition, and draft save occurs.
           Simply add new mail to Drafts folder as a "draft mail"(call Offset=A1), as new mail is added to Inbox by mail download.
      2. Upon next draft save,
          Add new draft mail data to Drafts folder (at Offset=A2),
          Delete previous version of draft mail at Offset=A1.
      3. Repeat step 2 by auto-save.
          Add new draft mail data to Drafts folder (at Offset=A<N>),
          Delete previous version of draft mail at Offset=A<N-1>
      4. Send mail
          After successful send, delete last version of draft mail at Offset=A<N>)

There are some known cases which can produce phenomenon of "'previous version of draft mail in local Drats folder' is not deleted".
And, (a) some of them are 100% reproducible, and (b) some of them are easily reproduced even though it's not 100% reproducible, and (c) some of them are impossible or pretty hard to reproduce problem consistently.
Biggest problem in this bug is: No one says "his problem is same as (a) or (b) or (c) or completly different problem from (a)/(b)/(c)" :-)
WADA:  When, How, did you find "10 or more copies"?
Taber:  When I have one or more draft emails open for a long time (hours), particularly if I have put the (windows) system through a suspend-resume cycle a couple of times.  It gets really funny if I have 3 mails in draft status for a day or so...I can get dozens of un-cleared partial drafts.

WADA:  Was status of Drafts folder normal and actually healthy(no outdated msf condition, Size of Drafts.msf is not ZERO) before you started mail composition or draft mail editing?
Taber:  This happens just infrequently enough that I don't monitor this.  I suppose if I were perfect I would.

WADA:  No interfere on Tb's Drafts.msf file open by other software? (You can easily check it by monitoring tool for file access. For example, Process Monitor if MS Win, opensnoop if Mac OS X)
Taber: As this condition happens more often when I have the folder encrypted, this is a windows kernel function that is not specifically monitored by ProcMon.  I know of no other software that is touching this directory.  Why, then, do I have encryption on?  This is a laptop, and for client data security I am required to encrypt all email folders.
Fred,

Based on my experience TB autosave does not cause this problem.

Win 7 Pro 64-bit, TB 31.1.2, 4 POP accounts that check every 10 minutes (no IMAP), all accounts use the same mail folders (i.e. only one Drafts folder, one Inbox, one Sent, etc) stored in the default appdata location on the local C: drive, no junk filtering, no encryption, TB autosave set to 1 minute, no automated backups, automatic TB folder compaction whenever 1 MB will be saved, "Allow Windows Search to search messages" and "Enable Global Search and Indexer" are both checked, the system is active with no suspension or hibernation involved.

I get one file left in drafts maybe a couple of times a week. I check my Sent folder to make sure the email was sent and then I delete the one left in Drafts. Once every few months I may have dozens for the same email left in Drafts. I have not noticed the trigger. The one left in Drafts a couple of times a week is of an email I had open composing for over a half hour. I think the longer I have an email open composing, the more likely the problem is to happen. If autosave (which happens every 1 minute) were the cause, I think I'd have more than one copy left in Drafts. The problem seems more likely to happen when my email contains one or more embedded images. This is not a big problem to me. I just spoke up thinking my setup and experience of the problem might help your analysis.

Jon
(In reply to WADA from comment #94)
> (In reply to Fred Stluka from comment #90)
> > Does your scenario explain why I would get 10 or more copies in Drafts,
> > when it takes me 15 minutes to compose an email message?
> 
> ... Good explanation about scenario that keeps failing to delete 
> previous draft ...

That explanation makes sense.  Thanks!

> When, How, did you find "10 or more copies"?
> Was status of Drafts folder normal and actually healthy(no outdated msf
> condition, Size of Drafts.msf is not ZERO) before you started mail
> composition or draft mail editing?

It's hard to be sure.  When I've seen the problem, it has been because
I noticed a high message count on the Drafts folder in the GUI, and 
opened it to find hundreds of messages left there from the previous 
several days, including 10+ copies of some messages that I took a 
while to compose.  I never looked at the Drafts and Drafts.msf files 
on disk to see what size they were, or whether status bits were set
internally.

> No interfere on Tb's Drafts.msf file open by other software?

Not that I know of.

> (You can easily check it by monitoring tool for file access. For example,
> Process Monitor if MS Win, opensnoop if Mac OS X)

For the past couple days, I've been keeping the Drafts folder open 
in a tab of the GUI, so I can more easily check it.  Sometimes I've 
even kept that tab open while composing in another window, and can
see it poking out from behind.  In these couple days, the problem
has NEVER occurred.  Perhaps it is somehow fixed?  Or perhaps having
the Drafts folder open prevents or silently fixes the problem?

--Fred
(In reply to WADA from comment #95)
> (In reply to Fred Stluka from comment #92
> > Or did I minunderstand?
> 
> You didn't misunderstand.
> ... Good explanation of typical autosave/delete behavior ...

Yeah, that's how I would expect it to act.  And that fits with 
the behavior I've usually seen for the past several years.
Thanks!

> ... Good explanation of how reproducible the bug is in various cases ...
>
> Biggest problem in this bug is: No one says "his problem is same as (a) or
> (b) or (c) or completly different problem from (a)/(b)/(c)" :-)

Yeah, as a programmer for 30+ years, I feel your pain.  I hate 
intermittent bugs!

--Fred
(In reply to Jon Maloney from comment #97)
> Win 7 Pro 64-bit, TB 31.1.2, 4 POP accounts that check every 10 minutes (no
> IMAP), all accounts use the same mail folders (i.e. only one Drafts folder,
> one Inbox, one Sent, etc) stored in the default appdata location on the
> local C: drive, no junk filtering, no encryption, TB autosave set to 1
> minute, no automated backups, automatic TB folder compaction whenever 1 MB
> will be saved, "Allow Windows Search to search messages" and "Enable Global
> Search and Indexer" are both checked, the system is active with no
> suspension or hibernation involved.

Good list of details!  Similar to mine:

- Mac OS X 10.6.8 (Snow Leopard)
- TB 31.1.2
- 7 POP accounts that check every 2 minutes (no IMAP)
- Each account uses its own folder tree (separate Drafts folder,
  separate Inboxes, separate Sent folders, etc.), but I have 
  150+ filters on my primary account moving incoming messages
  to various folders inside its Inbox folder, and I have a 
  couple of filters on other accounts that also move incoming
  messages to folders in the Inbox of my primary account.
  No single folder is ever written by more than one account 
  however.  In any case, none of the filters have anything to
  do with the Drafts folder.
- Mail profile (folder tree) is not in the default location.
  It is redirected by profiles.ini as described earlier.  
- Mail profile is on the primary local hard drive
- No junk filtering
- No encryption
- TB autosave set to 1 minute
- Automated backups via Mac Time Machine to wireless Time Capsule
- TB folder compaction whenever 1 MB will be saved, but always 
  prompt me first, and I only allow it when no mail is being 
  pulled or composed.
- Allow Spotlight to search messages = yes
- Enable Global Search and Indexer = yes
  - Hmmm...  This is another async process that could be accessing
    the Drafts folder, like backups do.  Related?
- I never close the lid or otherwise allow it to go to sleep 
  without explicitly closing TB first (because I wondered once 
  a while back if that was causing a different TB problem and 
  got into the habit of being careful just in case).


> I get one file left in drafts maybe a couple of times a week. I check my
> Sent folder to make sure the email was sent and then I delete the one left
> in Drafts. Once every few months I may have dozens for the same email left
> in Drafts. I have not noticed the trigger. The one left in Drafts a couple
> of times a week is of an email I had open composing for over a half hour. I
> think the longer I have an email open composing, the more likely the problem
> is to happen. If autosave (which happens every 1 minute) were the cause, I
> think I'd have more than one copy left in Drafts. 

Yeah, I've had occasional glitches like that for a year or more, 
but it was never enough of a problem to worry about.  Like you, I'd
notice the Draft, make sure I had remembered to send it, and then
delete the Draft.  Yeah, typically a message I'd taken a long time
to compose, especially if I changed the subject line or recipients
partway through.  I never got multiples though, just one per message.

More recently, I suddenly started to see multiple per message, that
probably ARE one per minute, so autosaves are not deleting old drafts.
And it suddenly started happening for lots of messages, perhaps even 
all of them that I sent that had a chance to create even a single 
draft.  I'd have hundreds of drafts in a day or so.

> The problem seems more
> likely to happen when my email contains one or more embedded images. 

Not a factor for me.  I never email images.  I always post them 
to my Web site and email a link to them.  I like to keep my email
storage as small as possible, since I have copies of every message
I've sent or received (except spam and mailing lists) for the past
30 years.  I hit delete often, but I never empty the Trash folder.
I just occasionally move all of the messages from Trash to a 
separate folder named Trash_<date>.

> This is not a big problem to me. I just spoke up thinking my 
> setup and experience of the problem might help your analysis.

Yeah, useful info.  May help the Mozilla guys to narrow it down.
Intermittent bugs like this, especially those based on a race
condition between async events (autosaves, backups, virus scans,
indexers, etc.) are a real pain to debug.

Meanwhile, as I said above, it has stopped happening completely
in the past couple days.

Thanks!

--Fred
(In reply to Fred Stluka from comment #98)
> For the past couple days, I've been keeping the Drafts folder open in a tab of the GUI, so I can more easily check it.
> Sometimes I've even kept that tab open while composing in another window, and can see it poking out from behind.
> In these couple days, the problem has NEVER occurred.
>  Perhaps it is somehow fixed?  Or perhaps having the Drafts folder open prevents or silently fixes the problem?

If problem is caused by "Tb's Drafts.msf file open by someone(including Tb's other tasks) while Tb's draft save task doesn't open Drafts.msf", problem will be avoided by "Tb keeps Drafts.msf open(write mode)". IIUC, "showing Drafts folder content at a Tab of a Messenger Window" will keep Drafts.msf open.

"Open/Close of MsgDB==xxx.msf by Tb" can be traced by NSPR logging with "timestamp,MsgCopyService:5,MsgDB:1". MsgCopyService:5 is for knowing "when save to Drafts folder was requested". If you monitor "Drafts.msf and Drafts file access" by monitoring tool at same time, it'll be helpful.

(In reply to Jon Maloney from comment #97)
> TB autosave set to 1 minute,
(In reply to Fred Stluka from comment #100)
> - TB autosave set to 1 minute

Default of auto-save interval == 5 minutes.
"one draft save every one minute while composing a mail" is perhaps a reason why both of you can see problem so frequently, can see so many non-deleted draft mails.
Check "[x] Show confirmation dialog when messages are saved" of Copies&Folders setting of each Identity. By dialog which is shown when draft save completed, you can know when draft save happened.
My current system config (more complete than in my previous posts...this config has evolved over the 5 years since I first reported this bug):
* Win 7 pro 64 bit
* Virus & malware scanner:  Microsoft Security Essentials
* Windows indexing:  on for the folders where mail is kept
* Windows encrypted file system:  on for the folders where mail is kept
* Backup:  entirely manual, never running when symptoms occur
* Mirroring/RAID:  none
* All drives directly attached, never disconnected (SATA)
* T'bird:  current version with Lightning (calendar) and 15 other plug ins (mostly irrelevant)
  - Glodaquilla (indexing enhancement)
  - gContactSync
  - Automatic export (of calendar)
  - Gloda indexing on
  - automatic compacting on
  - junk detection on
  - 50 filters spread across the accounts, none of them read/write the drafts folder
* Mail is 10 accounts, all POP, all automatically poling servers at different intervals (on purpose)
  - files not in standard location on local drive
  - all accounts point to same Local Folders set (draft, sent, junk, etc.)
  - auto-save drafts every 5 mins
  - AS OF NOW, PER WADA am keeping the Drafts folder "always open" as a separate tab

Appearance of symptom (repeating just for completeness):  
  - intermittent, no obvious correlation with usage (I've been trying to find one....)
  - 95% of the time no embedded images
  - is worse with long-open draft messages, particularly if more than one is being edited
  - is worse with suspend/resume
  
Possibly-related symptom:  over a several-month period, all the MSF files in seldom-used mail folders will be automatically deleted by something.  All the other (non-MSF) files are left unmolested, however.

Other apps running on same system:
  - the only user-level app open all the time is Firefox
  - about 120 win processes running
(In reply to David Taber from comment #102)
> Possibly-related symptom:  over a several-month period, all the MSF files in
> seldom-used mail folders will be automatically deleted by something.
> All the other (non-MSF) files are left unmolested, however.

If such issue can occur in your environment, why still no report about "who accesses Tb's Drafts.msf file" from you?
It's pretty easily known merely by getting Process Monitor log for one day only.
And "how long does it take to read entire Drafts.msf file for Drafts folder open", "how long does it take to write back Drafts.msf file for Drafts folder close", are also known by Process Monitor log.
And, if you get NSPR log with "timestamp,MsgDB:1", you can know "when Tb opened Drafts folder" and "when Tb closed Drafts folder".
Note: 
Bug 905576 occurs within Thunderbird only. No other process accesses Tb's .msf file and LocalStore file.
Problem perhaps occurs in race condition between "MsgDB open" and "MsgDB close",  and/or between "MsgDB open" and "MsgDB open".
"Why problem occurs with Network File only" is perhaps "if network file, it takes long to read entire xxx.msf for folder open".

I recommend you to get Process Monitor log and NSPR log with minimum environment, with special condition.
1. New Tb profile, with dummy news account only(Drafts of Local Folders is sufficient for test).
    If needed, create a dummy SMTP server. Disable auto-compact. Disable auto-save.
    Enable Encryption of all files under Mail directory or Profile directory, as done in daily use profile.
    Create 2 draft mails(call draft#0-1, drafts#0-2). Call file size of Drafts file "Size#0" (= size of draft#0-1 + size of drafts#0-2).
    Click other folder than Drafts(to not open Drafts upon restart), and terminate Tb.
2. Start Process Monitor, Restart Tb with "timestamp,MsgCopyService:5,MsgDB:1". 
3. Delete Drafts.msf file and re-create Drafts.msf file(force file size=ZERO)
4. Compose some mails, save as draft several times. (change subject upon each save, for ease of analysis)
    I don't know msgDatabase.summaryValid=false is set or not.
    IIRC, even if false is set, msgDatabase.summaryValid=false is reset by filter move, draft save etc., but I'm not sure.
5. End mail composition, terminate Tb, keep backup of log.
    I don't know msgDatabase.summaryValid=false is written in Drafts.msf  or not.
6. Restart Tb with NSPR logging enabled.
7. From context menu of Drafts, Compact.
    IIRC, if msgDatabase.summaryValid=false is set, or if "outdated msf condition" is raised,
    Compact invoked Rebuild-Index, but I'm not sure.
8. Open Drafts folder by folder click at Folder Pane.
9. "Repair Folder" of Drafts folder.

This is test for a variant of bug 261419 / bug 495760, with intentionally generated "msf file size=0" status.
Phenomenon is probably one of next.
-  No draft#0-1, drafts#0-2" in Drafts folder, if "Drafts.msf file size=0" is considered "msf file with no MsgDBHdr".
-  Non-deleted old draft, if "Drafts.msf file size=0" is considered "outdated msf condition".
-  draft#0-1, drafts#0-2, are kept, no non-deleted old draft mail,
    if expunged bit=On is set in X-Mozilla-Status: header and if Rebuild-Index is invoked by Compact request..
I haven't had any time to set up any test environments or special configs, sorry Wada.

However, what I have done is kept the Drafts folder "open" as a separate tab in the UI, and have done this for other folders where MSF files were getting wiped out inexplicably.

It's only been about a month, but I've done everything I can think of to cause the symptoms I was experiencing...and so far, the keep-the-folders-open tactic seems to avoid the issue.  I'll keep things set up this way and will report back in a few months.
(In reply to David Taber from comment #104)
> However, what I have done is kept the Drafts folder "open" as a separate tab
> in the UI...

Yeah, me too.  Seems to work perfectly if I keep the Drafts tab
open all the time.  Haven't yet taken the time to close it and 
see if the problem recurs.

--Fred
I was having the same problem, apparently it is a syncing issue with saving it to your email inbox in the cloud then Thunderbird re-downloading every version saved in the cloud back into the folder locally. You can try to save drafts locally instead. See the following link for full solution. This worked for myself.

https://support.mozilla.org/en-US/questions/1028205
FWIW I had been keeping the drafts folder open all the time (as a separate tab) and for months there was no issue.
Since jan 1, I have been keeping the drafts folder closed, and at least once a week I will get a boatload of spurious partial drafts that were saved, but not erased on send.  Nothing has changed in my hardware/software configuration other than windows and T'bird updates.
Files are all saved locally, so this is separate from the cloud-synch issue hypothesized by Faisal R.
WADA--sorry, but I have zero time to do any testing due to work deadlines.  Will resume test cases in April.
FWIW - Same bug exhibited in Seamonkey in my local Drafts folder, starting ~Feb 5, after updating to new version in Jan/Feb 2015.  Seems that the deletion/replacement of auto save versions of a message is now broken (again) - Similar bug circa 2009..  Autosave occurs after 5 minutes.  Submitted Seamonkey bug against this bug, as a duplicate?
aceman, does it sound like it lost track of the pointer to the draft message?
(note this only happens for Local Folder)
Flags: needinfo?(acelists)
If this wasn't so old, I'd blame some of the bugs for the "edit as new"/"edit as draft" where we really removed some "pointers"/ids to the draft message, exactly to decouple the new composition from the old one (so that 2 are in the drafts folder).

However the later comments saying about lost .msf files, that looks like the bug Jorg is observing. That a special link in a draft message or a sent message will cause TB to wipe drafs.msf which MAY have that effect of all the old deprecated drafts to become visible again.
That was bug 1216914, which I think is fixed. So maybe a test on current trunk would be useful here.
Flags: needinfo?(acelists)
aaarrggh, unbelievable that this still exists after 7 years... it is related to autosave some way and maybe inline images in edited message. No chances of debug logging from some developer who has functional & compilable Thunderbird sources?
I am closing this bug as a duplicate of bug 1216914.

In general, every saved or auto-saved draft remains as superseded in the Drafts folder until this folder is compacted.

There has been a bug that caused Drafts.msf (or in fact any other .msf file) to get deleted under certain circumstances. That caused all the superseded intermediate drafts to spring back to life when the .msf file was reconstructed from the raw message file.

This bug has been fixed and the fix will ship in TB 47 beta and will possibly also be included in an ESR TB 45.x release.

If there is still another problem, please raise another bug since this one has become way too confusing.

Kent, please note, I was not the first one to come across the missing Drafts.msf.
Status: NEW → RESOLVED
Closed: 8 years ago
Flags: needinfo?(rkent)
Resolution: --- → DUPLICATE
Jorg,

I think this ticket should not have been closed.

It describes a very specific and repeatable problem, with a very specific 
and reliable workaround, and it has not yet been fixed.  

The original problem description is still accurate:

> Several copies (of increasing length) of a draft message are kept in the 
> Drafts folder, even after the message has been fully composed and sent.
> 
> This is a regression from previous releases.
> 
> Reproducible: Always
> 
> Steps to Reproduce:
> 1.Write an email
> 2.Take a long time doing so
> 3.
>
> Actual Results:  
> Several copies of the draft remain in the DRAFT folder, even after the mail has been sent.
> 
> Expected Results:  
> Only one copy of the draft message at all times, and once it's been sent it's expunged.

Also, when I try to access the bug that it is marked as a duplicate of:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1216914

I get an error message saying:
- You are not authorized to access bug 1216914.

Should I now submit a new ticket, and should all the 29 people on the 
CC list now add themselves to that ticket?  Seems like it would be 
easier to just leave this one open until it is fixed.

More info (summarized from my posts above):
- Always happens to me, unless I leave the Drafts folder open in a tab,
  in which case it never happens.
- I'm always on the latest version of TB, allowing it to upgrade when
  it asks.
- I use POP3, not IMAP
- Not related to Local Folders.  I don't use Local Folders.
- When I keep the Drafts folder open, I can see each new auto-saved
  draft replacing the previous one as I type a long message, and when
  I finally hit send, I see the last draft get deleted.  If I close 
  the Drafts folder tab, each new draft leaves the old one behind, 
  so they accumulate, and when I send, they all remain.
- Not related to compacting folders.  I explicitly control when I 
  compact and never do it while composing.
- Seems unrelated to OS (Mac, Windows, etc.)
- Seems unrelated to file system attributes (compression, encryption,
  ACLs, etc.)
- Not related to Time Machine or other backup software locking files.
  I have turned off my Time Machine backups and it still happens.
- Started to happen to me when I moved from TB 31.0 to 31.1.2
- Note related to embedded images.  I never use them.

What can I do to help get this resolved?

Thanks!
--Fred
The other bug 1216914 is a security issue, so it is access restricted.

What got fixed is that Drafts.msf *occasionally* got removed, so superseded messages sprang back to life.

You are basically saying that unless you keep the drafts folder open in a tab, *all* superseded drafts *always* persist? I've never seen this before and I'm using a POP3 setup with autosave every minute.

Can you please confirm the *all* and *always* part. If it was occasionally, that I believe it's fixed.

If you indeed *always* keep all saved/auto-saved drafts, then there is something wrong with your system.

So please clarify whether we're talking the *occasional* case, which was fixed or the *always* case which needs further investigation.

P.S.: Since this bug was talking about "missing Drafts.msf" and got way too confusing, it was a good idea to make it a duplicate of the other bug. We can open a new bug with a short description and focus on the precise problem.
(In reply to Jorg K (GMT+2, PTO during summer, NI me) from comment #114)
> The other bug 1216914 is a security issue, so it is access restricted.

OK.  Interesting problem.  I often close tickets on my project as 
duplicates, but the other ticket is always accessible.  For security
reasons, I can see why you'd want to hide the other ticket, but that
makes it less appealing to have this ticket closed, since any notes,
explanations and workarounds may be in the other ticket.  If you have 
time, it might be a good idea to copy the relevant snippets from that
ticket here, at least the parts without any sensitive security info.
Low priority perhaps, though, if the fix for this problem is going to 
be released soon.

> What got fixed is that Drafts.msf *occasionally* got removed, so superseded
> messages sprang back to life.

Interesting...  I've had another problem recently that might be 
explained by that.  Mail folders frequently (a couple times per 
day) lose their settings for whether to show messages threaded, 
which columns to show, etc.  Would be explained by *.msf files 
being deleted.

Probably related is a third bug I've seen.  Sometimes my filters 
refuse to move an incoming mail message to a folder.  Instead, I 
get a popup error message about being unable to write to the 
folder.  Such folders also fail to show their count of unread
messages in the folder tree view.  My fix is to click on the 
folder, which causes the count to appear and the filters to start
working again.  Clicking on a folder with no *.msf file causes
the *.msf file to be re-created, so this would also be explained 
by *.msf files being deleted.

I look forward to the new release.  It may fix a LOT for me.

Meanwhile, since you pointed this out to me, I set up a script to loop
once per second showing the number of *.msf files in my mail tree.  
I've left it running for about a day now.  When I fire up TB, it 
shows 640.  Then immediately, or after several minutes, the count 
starts dropping.  Slowly, no more than one per second, my *.msf files
are deleted, until the count drops to 370-380 or so.  Then the 
deletions seem to stop.  If I close TB meanwhile, the count remains 
steady.  It is definitely only happening while TB is running.  If I 
restart TB, the count jumps immediately back up to 640, and soon 
starts dropping again.  

Does this seem related to the bug you've just fixed?

> You are basically saying that unless you keep the drafts folder open in a
> tab, *all* superseded drafts *always* persist? I've never seen this before
> and I'm using a POP3 setup with autosave every minute.

Well, I'm not sure.  It was occasional in Oct 2014, when I first reported 
it by commenting on this ticket.  Started suddenly when I moved from 31.0 
to 31.1.2, and gradually got worse.  Since Nov 2014, I've been keeping the 
Drafts folder open in a tab, and had no problems.  But whenever I 
accidentally close the tab, I soon notice the problem again.

> Can you please confirm the *all* and *always* part. If it was occasionally,
> that I believe it's fixed.

OK.  Here's hoping!...

> If you indeed *always* keep all saved/auto-saved drafts, then there is
> something wrong with your system.
> 
> So please clarify whether we're talking the *occasional* case, which was
> fixed or the *always* case which needs further investigation.
> 
> P.S.: Since this bug was talking about "missing Drafts.msf" and got way too
> confusing, it was a good idea to make it a duplicate of the other bug. We
> can open a new bug with a short description and focus on the precise problem.

OK.  Let's leave this closed for now.  Once I move to a fixed version, I'll 
add a comment here if the problem is still there.  Any idea when that will 
be?  I'm currently running 45.1.1 on the "release update channel".

Thanks!
--Fred
(In reply to Fred Stluka from comment #115)
I CC'ed you on the other bug, so you can read it.

> Interesting...  I've had another problem recently that might be 
> explained by that.  Mail folders frequently (a couple times per 
> day) lose their settings for whether to show messages threaded, 
> which columns to show, etc.  Would be explained by *.msf files 
> being deleted.
Yes, when .msf files get deleted, you lose these settings.
Let's get things straight here: I lost my drafts folder about once a month, so that's why it took me seven months to track down the problem. .msf files should not be get lost by the dozens, that's just wrong. What I fixed was a very special case.

> Clicking on a folder with no *.msf file causes
> the *.msf file to be re-created, so this would also be explained 
> by *.msf files being deleted.
Yes, clicking on the folder will recreate the .msf, in case of a draft folder many superseded drafts will come back to life.

> I look forward to the new release. It may fix a LOT for me.
It's not going to be released officially very soon. If you want to try it out, run an Earlybird version TB 49 from here:
http://ftp.mozilla.org/pub/thunderbird/nightly/latest-comm-aurora/thunderbird-49.0a2.en-US.win32.installer.exe
 
> Meanwhile, since you pointed this out to me, I set up a script to loop
> once per second showing the number of *.msf files in my mail tree.  
> I've left it running for about a day now.  When I fire up TB, it 
> shows 640.  Then immediately, or after several minutes, the count 
> starts dropping.  Slowly, no more than one per second, my *.msf files
> are deleted, until the count drops to 370-380 or so.  Then the 
> deletions seem to stop.  If I close TB meanwhile, the count remains 
> steady.  It is definitely only happening while TB is running.  If I 
> restart TB, the count jumps immediately back up to 640, and soon 
> starts dropping again.  
> 
> Does this seem related to the bug you've just fixed?
That seems absolutely terrible. There is something wrong on your system. It is correct that .msf files get constantly opened, updated and closed. But their count shouldn't change unless you delete or create a folder.
Using "Everything" on Windows, I can see that I had 3643 .msf files on the system, I deleted a folder, so now I have 3642. That count doesn't change regardless of whether I start or close TB. Are you sure you're counting them correctly? Are you sure your anti-virus doesn't cause damage. Seriously, I've been using TB since 2010 and apart from bug 1216914 I've never lost one.

> OK.  Let's leave this closed for now.  Once I move to a fixed version, I'll 
> add a comment here if the problem is still there.  Any idea when that will 
> be?  I'm currently running 45.1.1 on the "release update channel".
As I said, use an Earlybird as indicated above. The upcoming beta TB 47 will also have the fix. But for that you'd have to wait about a week.
Hi all,
Regarding the mass disappearance of msf files, I have a similar symptom that occurs **only when the msf files are stored in encrypted folders managed by windows' EFS**.
I suspect this is a conflict between EFS' housekeeping functions and T'Bird's file locking of msf files.
You can see the drivel I wrote over the years in this bug thread.
I am in the "I give up" stage on this one.  We'll see if the various upcoming releases fix it...
Any reproducible mass disappearance of should be reported in another bug.

One aspect of this bug here is about problems with duplicated drafts, most likely due to a disappeared .msf file, and that bug is fixed. So I closed this bug here by making it a duplicate of the bug which fixed this problem. If we mix various issues into the same bug, we will never be able to address them all.

Further to my comment #116: I'm still counting .msf files on my system with the Windows utility "Everything" which reads the MFT, and my count of 3462 hasn't changed since yesterday despite opening and closing TB many times and even installing a different version.

So once again, if you have reproducible disappearance of those .msf files, please report another bug. Those .msf files are important for the proper functioning of TB and should not disappear or be touched by other software.
Flags: needinfo?(rkent)
Thanks for the "everything" tip...didn't know about that one.
FWIW, the **count** of msf files on my system doesn't change, but their size sure does.  When my msf files get goofed, they're all still there, but they are of 0 bytes length.
If I went through all my file trees and created all the indexes (I have done this a few times), the MSF files would stay in tact for a while.  But after a few weeks, 10% of them would be zero-length, then 20%, then 30%...eventually, over half of them are zero length even though I haven't done anything with those underlying mailboxes (nothing new going in, nothing even being read by me).
It's data rot caused by some evil background process interaction...and it only occurs with the combination of T'bird and EFS storage on local drives.
Yes, 0 byte .msf happen as part of the disappearance. First it gets deleted and then it gets recreated with 0 bytes. I've checked my .msf files. There were a handful, all left-overs from folder deletion. So I deleted those. Now all my 3600+ .msf files are non-empty. I will keep an eye on them with "Everything".

So if you're saying that it only occurs with EFS at work, we should raise another bug for that. It may well be that TB doesn't handle "file locked" well when trying to access the .msf.
OK, I'll open a bug with that.  Narrow use case, but for anyone interested in information security a biggie.
OK, 7 years after filing the original bug...here's the new "pure" instance.

https://bugzilla.mozilla.org/show_bug.cgi?id=1279344
(In reply to Jorg K (GMT+2, PTO during summer, NI me) from comment #116)
> That seems absolutely terrible. There is something wrong on your system. It
> is correct that .msf files get constantly opened, updated and closed. But
> their count shouldn't change unless you delete or create a folder.
> Using "Everything" on Windows, I can see that I had 3643 .msf files on the
> system, I deleted a folder, so now I have 3642. That count doesn't change
> regardless of whether I start or close TB. Are you sure you're counting them
> correctly? Are you sure your anti-virus doesn't cause damage. Seriously,
> I've been using TB since 2010 and apart from bug 1216914 I've never lost one.

Yes, I'm sure I'm counting them correctly.  It's a one-liner in Unix:
% find ~/fred/Mozilla/TBird/Profile/mail -name \*.msf | wc -l
I'm on a Mac, running OS X 10.11.2, which is based on BSD Unix.
I have that line in a script and another script loops calling it once
per second.  Lately, I'm leaving the Drafts folder closed, and it stays
at 640 for a while, but after several minutes, starts dropping gradually
to about 370-380.  

I'm 

I'll wait for release 47.  If this and/or my other problems persist 
after that, I'll open new tickets.

Thanks!
--Fred
(In reply to Jorg K (GMT+2, PTO during summer, NI me) from comment #116)
> ...Are you sure your anti-virus doesn't cause damage.  Seriously,
> I've been using TB since 2010 and apart from bug 1216914 I've never lost one.

I'm not running any anti-virus software.  Mac, not Windows, and I'm pretty
careful about where I browse and pretty savvy about not allowing any 
installs.  Don't use MS Word, Excel, or Outlook, so I'm not vulnerable
to those types of macros.

Like you, I've been using TBird a long time.  Over 10 years.  Moved to 
TBird 1.0.6 in Aug 2005 from Netscape Messenger 4.79.  Used Netscape for 
about 10 years before that.  I've never had *.msf files vanishing until 
recently.  Seems to have started around the time I moved from TB 31.0 
to 31.1.2 in Oct 2014.

--Fred
Let's continue the discussion in bug 1279344.
(In reply to Fred Stluka from comment #123)
> I'll wait for release 47.  If this and/or my other problems persist 
> after that, I'll open new tickets.
I wouldn't put my hopes too high. In bug 1216914 I fixed the sporadic deletion of a MSF file, mainly affected Drafts.msf (complained about here in comment #79, last line).
Bugs with ESF(exact case) in bug summary. Bug 1279344 was opened by David Taber on 2016-06-09.
Bug 881829  Indexing stalls indefinitely on Sent folder (activity manager) in POP with EFS file encryption enabled
Bug 1279344 msf files disappear/get emptied out when stored in an Windows EFS directory, also disappearing MSF files on Mac.

A known issue in "writing sent mail copy to local Sent folder".
  Without explicit Sent Folder open, Tb adds data to Sent folder.
  (explicit open in this context = Folder open by folder click at folder pane, which invokes Rebuild-Index if .msf is broken.)
  ("append mail data to file named Sent" may occur even when Sent.msf file is not normally opened.)
IIRC, this is similar in "draft mail adding to Drafts folder" and "old draft deletion from Drafts folder".

This bug was opened on 2009-03-11 by David Taber, and following is Comment #55 by David Taber on 2014-03-23 which is first referring to EFS in this bug.
> I've been using NTFS with the EFS and file system indexing feature for 10 years on XP, Vista, and Windows 7, all pro, all 64-bit.

To Jorg K.
Did problem of bug 1216914(you fixed, and invisible for us, and this bug was duped to it) start to occur from 2009 or before?

Following is a part of Comment #124 by Fred Stluka on 2016-06-10.
> I've never had *.msf files vanishing until recently.
> Seems to have started around the time I moved from TB 31.0 to 31.1.2 in Oct 2014.

To Jorg K.
Was problem of bug 1216914 generated in 2014?
FYI.
Bug 905576 is for phenomenon of "xxx.msf is somehow deleted or fileSize=0 of xxx.msf is set while Tb is normally running -> Rebuild-Index is invoked upon next explicit folder open" when local mail folder file is located on SMB file share server.
It looked for me that difference between "local mail folder file on local HDD" and "local mail folder file on SMB server" was "time required to open/close xxx.msf file and xxx file".
If "EFS for loacl mail folder file" is cause of frequent loss of Sent.msf, Drafts.msf etc. in your environment, it may be that "time required to open/close xxx.msf file/xxx file in EFS" is far bigger than "file on SMB server".
Big different between Sent and Drafts.
- Sent   : "Rebuild-Index upon folder open due to outdated msf condition or deleted msf file" is not so obvious for users.
- Drafts : "Rebuild-Index upon folder open due to outdated msf condition or deleted msf file" easily/frequently produces
  "many undeleted old draft mails in Drafts folder". So, problem can be frequently detected by users.
(In reply to WADA from comment #127)
> To Jorg K.
> Did problem of bug 1216914 (you fixed, and invisible for us, and this bug was
> duped to it) start to occur from 2009 or before?
This problem was a regression of bug 402392
http://hg.mozilla.org/comm-central/rev/097bc36f180d#l47.485
landed at Christmas 2011. Nice Christmas present to make MSF files disappear :-(

For what it's worth, I haven't seen this bug in years.
I originally opened it 13 years ago.

You need to log in before you can comment on or make changes to this bug.