Open Bug 558387 Opened 14 years ago Updated 2 years ago

Offline Storage is not automatically compacted with "Mark as deleted" model. Compact "fires" but folder not compacted.

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Platform:
x86
Windows 7
Type:
defect

Tracking

(Not tracked)

Status:
UNCONFIRMED

People

(Reporter: tech4only, Unassigned)

Details

(Whiteboard: [gs?][dupeme?][reporter gone]) 

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4

The offline storage mail folders could not be compacted automatically. It can only manually done by using "File"->"Compact Folders".

Reproducible: Always

Steps to Reproduce:
1. Delete all mail in Inbox
2. Restart Thunderbird

Actual Results:  
The file size of inbox.msf  is 0 (c:\users\[username]\appdata\roaming\thunderbird\profiles\[profilename]\imapmail\[imapaccount]).

However file size of Inbox does not changed.

Expected Results:  
If it is autosyn and auto-compact, the file size of Inbox should decrease, and the inbox should only have mail specified within the autosync_max_age_days.

There are the settings for the client. Changing the first two settings does not affect the result.

("mail.imap.use_stats_for_biff",false)
("mail.server.default.autosync_offline_stores",true);
("mail.server.default.autosync_max_age_days",2);
("mail.prompt_purge_threshhold",true);
("mail.purge_threshhold",100);
("mail.imap.expunge_after_delete",true);
("mail.imap.cleanup_inbox_on_exit", true);

We use roaming profile, and would like to control the file size of mailbox.
So you are saying that auto-compaction doesn't fire ?
Component: General → Backend
Product: Thunderbird → MailNews Core
QA Contact: general → backend
Auto-compaction does fire. Sometime I could see briefly something like "..compacting..." if I deleted a lot of mail from inbox. However the offline storage files are not touched by the auto-compaction.
David is this by design ?
Autocompaction only fires if you turn it on - tools | options | advanced, network and diskspace, automatically compact folders when it will save XX KB (though I'd suggest a fairly large number here, like 1000KB, i.e., 1MB).
Autocompaction is on. It can be verified from the GUI, or the two settings in the preference file as included in the original post.
("mail.prompt_purge_threshhold",true);
("mail.purge_threshhold",100);
Quick check result with Tb 3.0.4, with Gmail IMAP.
(auto compact setting)
> mail.purge.ask=true
> mail.prompt_purge_threshhold=true
> mail.purge_threshhold=10 (10KB)
(local mail folder)
  A1 : total mail data size = SZa KB (> 10 KB, threshold)
(IMAP folder)
  B1 : offline-use=on, 
       mail-1      = SZ1 KB (keep in B1 to avoid big) 
       other mails = SZb KB
       total mail data size = SZ1 KB + SZb KB
  B2 : offline-use=off, empty

(Case-1: Move to Trash model, Case-2: Remove it immediately model)

(1) N times of { Move all mails except mail-1 in B1 to B2;
                 Move back all mails in B2 to B1;          }
    => offline-store size of B1 == SZ1 KB + (N+1)*SZb KB
    Note: mails to which \Deleted flag is stored are autoatically expunged
          at server because of Gmail IMAP.
(2) Restart Tb (to force auto-compact by step (3))
(3) Move all mails in local folder A1 to A2
    => Dialog before auto-compact => OK => Auto-compact is invoked.
(4) offline-store file size of B1 is reduced to (SZ1 KB + SZb KB).
(5) Move back all mails in A2 to A1 for next test.

(Case-3: Mark it as deleted model)

Offline-store file size was not reduced by above test, although mails to which \Deleted flag are automatically expunged at Gmail IMAP server by Gmail.
Because explicite "Compact" is needed to expunge mails of \Deleted flag when "Mark it as deleted" model, and because compaction of offline-store is invoked  after expunge command to server, it looks that compaction of offline-store file is not invoked by auto-compact if Mark it as deleted model.
tech4only@gmail.com(bug opener), which IMAP delete model do you use?

If ordinal IMAP server and "Mark it as deleted" model, mails to which \Deleted flag is stored are displayed with red strike line at thread pane. If auto-compact invokes expunge command in Case-3, mails with the red strike line is automatically removed from thread pane by auto-compact. I believe it shouldn't happen.
Separation of "expunge from server" and "compaction of offline-store" is required to avoid very expensive compact of big offline-store file upon each expunge request by user for small number of mails. I think such separation can be a solution of this bug.

By the way, tech4only@gmail.com, if you use "Mark it as deleted" model with Gmail IMAP, I recommend you to change to "Remove it immediately" model, because you are confused by auto-expunge of Gmail IMAP.
WADA, I am digesting your post and will check if I can duplicate your test.

For a quick response, the delete mode on my client is "Case-1: Move to Trash model". Also mail.purge.ask=true, but not dialog is prompted.
(In reply to comment #8)
> For a quick response, the delete mode on my client is "Case-1: Move to Trash
> model". Also mail.purge.ask=true, but not dialog is prompted.

Did you delete mail in local mail folder(POP3 or Local Folders)? Or IMAP folder?
I have no experience of dialog before auto-compact invoked by delete of mails in IMAP folder(I usually use Mark as deleted model for testing and I use IMAP for adhoc testing only). So, I deleted mails in local mail folder in my test to invoke dialog before auto-compact reliably.
It is all about deleting mail in IMAP. 

The current TB with offline storage behaves like POP. It does have a lot of benefits. However, we do like to control the size of the profile to ease roaming profile. This is why the autocompaction question is raised.

I did a few more tests with WADA's inspiration. This may shed some light.

1. Move mail from IMAP folder to a local folder, prompted to compact.
2. Move mail from local folder to local folder, prompted to compact.
3. Move mail from local folder to IMAP folder, not prompted.
4. Move mail from IMAP folder to IMAP folder, not prompted.
5. (Not related to auto compact) Delete an IMAP folder, the mbox file gone immediately from local file list.

In a pure IMAP environment, no local folder should be involved. So ideally, #3 should work.
tech4only, do you still see this problem when using version 3.1?

bienvenu, see comment 10 item 3
We stopped using TB after migrating to GMail, and I could not verify if it is fixed in the newer version.

Sorry not responding earlier. The message was overlooked.
wada, no dupes in a year. so do we still need this bug?
Summary: Offline Storage is not automatically compacted → Offline Storage is not automatically compacted with "Mark it as deleted" model
I think this bug should be kept open.
unclear if duplicate of an existing report. But I did come across a support posting where compact failed. Need to find it

FWIW compact + "mark as deleted" bugs https://bugzilla.mozilla.org/buglist.cgi?j_top=OR&f1=longdesc&list_id=7808085&short_desc=compact&v3=Markasdeleted&o1=substring&resolution=---&o2=substring&query_format=advanced&f3=longdesc&f2=longdesc&short_desc_type=allwordssubstr&v1=Mark%20as%20deleted&v2=Mark-as-deleted&product=MailNews%20Core&product=Thunderbird
Summary: Offline Storage is not automatically compacted with "Mark it as deleted" model → Offline Storage is not automatically compacted with "Mark as deleted" model. Compact "fires" but folder not compacted.
Whiteboard: [dupeme?] → [gs?][dupeme?][reporter gone]
This bug is perhaps;
 Because MsgDBHdr of "mail flagged \Deleted" is not removed when "Just mark it as deleted",
 expungedBytes of flder is not incremented by "mail delete" if "Just mark it as deleted".
 So global "total of expungedBytes" is not incremented.
 Becuse trigger of Auto-Compact is "total of expungedBytes > mail.purge_threshhold_mb"
 && "time after last Auto-Compact > one hour", "dialog before Auto-compact" is not
 ahown by "mail delete at IMAP folder".
 Because Compact = "issue EXPUNGE command" + "compact of offline-store file" in IMAP,
 and because Auto-Compact is invoked on "msgFolder of expungedBytes>0" only,
 "compact of offline-store file" is not invoked by Auto-Compact if Just mark it as deleted,
 even when Auto-Compact is triggered by other "mail delete" at local mail folder.

Following may be a solution.
 When EXPUNGED status of a mail is detected(by uid fetch 1:* (Flags) upon Mbox open,
 by unsol EXPUNGED responce via IDLE) and MsgDBHDR of the mail is removed by Tb,
 increment expungedBytes of msgFolder.
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.
FTI.
In bug for "Repeated download of all mails by auto-sync", followig is reported.
  file size of OfflineStore file is not reduced until manual Compact of the IMAP folder is executed.
Because Auto-Compact is enabled by default, I think Auto-Compaact is invoked sooner or later in usual environment. So, issue of "Auto-Compact doesn't invoke Compact of IMAP folder" probably exists.
"Expunge after each delete" may be relevant. 
And, as seen in test of comment #6, file size is not reduced if Just mark it as deleted(IIRC, bug exists for this issue).
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.