Closed Bug 546384 Opened 10 years ago Closed 10 years ago

another new nstmp file created and not deleted each time after using "Compact Button", but not when "File/Compact Folders" from menu - handle offline store streaming errors and track compact status to prevent multiple compacts

Categories

(MailNews Core :: Backend, defect, major)

x86
All
defect
Not set
major

Tracking

(thunderbird3.1 beta2-fixed)

RESOLVED FIXED
Thunderbird 3.1b2
Tracking Status
thunderbird3.1 --- beta2-fixed

People

(Reporter: tilman, Assigned: Bienvenu)

References

(Blocks 1 open bug)

Details

(Keywords: perf)

Attachments

(2 files, 1 obsolete file)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

I have put the "Compress" button in the toolbar. Each time I press it, another nstmp file is created in the imap folder, which is 
D:\Thunderbird\ImapMail\XXXX.xxxx.de . (I don't have enough space on C:). This effect does NOT happen if I compress all folders through the item in the file menu "Compress all folders of this account". Note that the german text for the compress button means something like "permanently remove deleted messages from the selected folder", i.e. it is a different functionality than the menu item.

I found this out only because I was wondering why my D: drive was getting full. I had nstmp files up to "nstmp-70", which were using over 10 GB!!!

I press "compress" often because I have thunderbird configured that messages are only marked to be deleted, not deleted immediately.

Reproducible: Always

Steps to Reproduce:
0. Delete something in an imap folder, so that it is marked for deletion.
1. Configure "compress" button in the toolbar (right click on the toolbar and drag it there)
2. in thunderbird, press Compress button
3. open imap folder in Windows explorer, see another nstmp file

Actual Results:  
another nstmp file appears

Expected Results:  
delete the nstmp file after compress is done

I suspect that the implementation for the "compress" button forgets to clean up.
Possibly similar bug here:
https://bugzilla.mozilla.org/show_bug.cgi?id=544843
Problem was partially reproduced with Tb 3.0.1 on MS Win-XP SP3.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

(1) Use [Gmail]/All Mail, offline-use=on, wait for end of auto-syc.
    [Gmail].sbd\All Mail : 11,552 KB, [Gmail].sbd\All Mail.msf : 3,829 KB
(2) Context menu of [Gmail]/All Mail, Compact => nstmp-N(nstmp if N=0) is used
(3) While compaction is running, Context menu of [Gmail]/All Mail, "Compact"
    => nstmp-(N+1) is created
(4) Repeat (3) multiple times => nstmp-(N+m) is creted
=> Phenomenon of multiple nstmp-X was observed.
(5) Wait for end of compaction => all of nstmp-X were cleanly deleted.
=> Garbages of multiple nstmp-X was not observed.

Do you enable "offline use" of the IMAP folder?
(Folder Properties/Synchronization)
If yes, how many mails do you have in the folder? What is file size of offline-store?

Does "Compact" of the IMAP folder never ends normally, even if you don't disturb comaction by "next comact request while previous compaction is running(file size of newest nstmp-N is increasing)"? 

Do you use auto-backup software for Tb's mail directory?
Do you run virus-scan periodically for files under Tb's mail directory?

AFAIR, "next comact request while previous compaction is running" didn't invoke next compaction with message like "compaction is runnning", "other process is in progress", etc. AFAIR, it was true during my testing with nightly of Tb3.0XNPre. Regression?
(In reply to comment #2)

Hello,

My bug happens when I do wait until done.

I just tried what you did, i.e. several compress without waiting. In that cases, all nstmp files are deleted correctly. However only one folder had something to delete. Then I deleted something in several imap folder, and then pressed compress quickly in all five of them. Two nstmp files were left, although not from all folders. The inbox folder is 200MB large, and "his" nstmp file was properly cleaned up this time.

I do not use auto-backup software for Tb's mail directory.

I do have an antivirus product but don't know when it runs. It does have a realtime monitor, however.  (Corporate environment)

Offline use is enabled, see screenshot.
I have 1126 mails in the inbox folder, the file is 200MB large.

Tilman
I could see the problem with many compact requests of [Gmail]/All Mail only.
Interfere by "new mail check" and "mail move" were added to previous test.
(1) In addition to previous setups of Tb 3.0.1,
    Change number of cached connections to 5 (I set 1 in preious test)
    Enable "Check new messages at startup/every 1 minute",
    Enable "Check this flder for new messages" of [Gmail]/All Mail
(2) Move 2 mails in [Gmail]/All Mail to [Gmail]/Trash,
    5 to 10 times of "Compact" of [Gmail]/All Mail,
    Move 2 mails in [Gmail]/Trash back to [Gmail]/All Mail.
    5 to 10 times of "Compact" of [Gmail]/All Mail
(3) Repeat step (2) many times.
(4) File size of some of nstmp-X = 11MB (all mails are copied)
    File size of other nstmp-Y is less than 11MB, and is increasing.
(5) Wait for a while, all of nstmp-X becomes 11MB, CPU utilization of Tb=0%
    => 15 nstmp-X of file size=11MB were kept.
       (minimum X: nstmp(X=0), maxmum X: nstmp-20 in my test)

Problems looks next for me.
(Problem-1)
  If "Compact" is requested multiple times for an IMAP mail folder,
  "Compact" task is scheduled for each "Compact" request.
(Problem-2)
  Some of scheduled "Compact" tasks don't delete nstmp-X file,
  if something wrong is detected after end of copy to nstmp-X.
  ("replacing of .msf" is probably fails due to contention of ".msf")
Confirming based on our test results, although solid/crisp/mimimum problem recreation procedure is not found yet.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Change "Compress" in bug summary to "Compact"(term of en-US build) to avoid confusion.
Summary: another new nstmp file created and not deleted each time after using "Compress Button", but not when "Compress all folders" from menu → another new nstmp file created and not deleted each time after using "Compact Button", but not when "File/Compact Folders" from menu
regression?
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
1) One factor might be that this is an old PC. It is a Pentium 4 although with 3GB of RAM. The compacting of the inbox takes about 20 seconds.
2) Due to the last comment on https://bugzilla.mozilla.org/show_bug.cgi?id=544843 , I'd say that we both have the same bug.
(In reply to comment #9)
> The compacting of the inbox takes about 20 seconds.

A facter of it is probably Bug 539389. Even when local file, "write request per each mail data line" of Tb3 makes file copy(copy mail data to nstmp-X in this bug's case) time longer than Tb2.   

> Bug 544843, I'd say that we both have the same bug.

I think so, but I believe that your this bug has far better bug summary and far better observation of problem/additional test results by you than that bug.
Duplicate of this bug: 544843
I have a fix for this.
Assignee: nobody → bienvenu
Attached patch proposed fix with unit test (obsolete) — Splinter Review
This patch does several things:

1. Make compacting offline store handle errors trying to stream messages. Now we continue to the next message. Previously we would just abandon the compact attempt, leaving nstmp files lying around.

2. Make compacting offline stores hold onto the folder lock for the whole compact process instead of just grabbing it for every message. This prevents the user from getting two compacts going on, or autosync sneaking in and appending to the offline store during a compact. 

3. Make compact of an imap folder with an offline store only send a notification when both the online expunge and the offline compact are finished.  This is required to make a remotely reasonable unit test.

4. Added a test to the offline store compact unit test to try to test this. It's a bit tricky because it's essentially a race between the online expunge and the offline compact so I had to tweak the unit test to give us a better chance to lose that race.

I'd really like this to get some testing during beta 2. I'll also generate a try server build in a minute...
Attachment #436002 - Flags: superreview?(bugzilla)
Attachment #436002 - Flags: review?(bugzilla)
Andrey, this is where I'm working on a fix that might help with bug 555171 - try server builds I requested should show up in a couple hours here - http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry
Gerv, you might be interested in the above try server build, because you've had issues with compacting your imap folders&offline store.
I hope this is related to bug 555171, or rather, this fix will help with that bug, but this is the bug I was able to reproduce and fix.
(In reply to comment #14)
> Andrey, this is where I'm working on a fix that might help with bug 555171 -
> try server builds I requested should show up in a couple hours here -
> http://tinderbox.mozilla.org/showbuilds.cgi?tree=ThunderbirdTry

What exact file from there should I try? The last file from bienvenu@ dated 2010-03-15.
Andrey, there were errors in my try server build, so I need to fix them and request a new one.
Attachment #436002 - Attachment is obsolete: true
Attachment #436191 - Flags: superreview?(bugzilla)
Attachment #436191 - Flags: review?(bugzilla)
Attachment #436002 - Flags: superreview?(bugzilla)
Attachment #436002 - Flags: review?(bugzilla)
Comment on attachment 436191 [details] [diff] [review]
fix with header file changes

>-nsresult nsOfflineStoreCompactState::CopyNextMessage()
>+nsresult nsOfflineStoreCompactState::CopyNextMessage(PRBool &done)
> {

This should have a comment as to why it is potentially trying to copy more than one message.

>+    nsCOMPtr<nsISupports> thisSupports;
>+    QueryInterface(NS_GET_IID(nsISupports), getter_AddRefs(thisSupports));

I think this is better:

nsCOMPtr<nsISupports> thisSupports = do_QueryInterface(this, &rv);
Attachment #436191 - Flags: superreview?(bugzilla)
Attachment #436191 - Flags: superreview+
Attachment #436191 - Flags: review?(bugzilla)
Attachment #436191 - Flags: review+
(In reply to comment #20)

> >+    nsCOMPtr<nsISupports> thisSupports;
> >+    QueryInterface(NS_GET_IID(nsISupports), getter_AddRefs(thisSupports));
> 
> I think this is better:
> 
> nsCOMPtr<nsISupports> thisSupports = do_QueryInterface(this, &rv);

That doesn't compile, unfortunately.
fix checked in.
Status: NEW → RESOLVED
Closed: 10 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
Dominique (bug 544843), Tilman, are your issues related to this completely gone?

Andrey (bug 555171) is no longer using thunderbird, but I think we can safely dup his bug to this.  (does autosync issue in bug 555171 comment 9 sound like bug 541917/bug 531033?)
Severity: normal → major
Keywords: perf
OS: Windows XP → All
Summary: another new nstmp file created and not deleted each time after using "Compact Button", but not when "File/Compact Folders" from menu → another new nstmp file created and not deleted each time after using "Compact Button", but not when "File/Compact Folders" from menu - handle offline store streaming errors and track compact status to prevent multiple compacts
Target Milestone: --- → Thunderbird 3.1b2
I remember that one or two releases after "fix checked in", the bug no longer happened. In the meantime I have also moved to a faster PC. I just looked in my imap dir, and it did have a few leftover nstmp files - but they were a month old. So the only thing that is sure, is that I can no longer reproduce the error at will. (I am compacting the folders several times a day) So I'd consider it fixed. Mostly :-)
You need to log in before you can comment on or make changes to this bug.