499630 - "Compact" of IMAP folder never reduces offline-store file size (Tb Trunk, even after fix of Bug 420115, Bug 466730, Bug 465385. File/Compact Folders is required)

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[WADA:World Anti-bad-Duping Agency]

"Compact Folder" of Gmail IMAP folder never reduces offline-store file size. (folder other than [Gmail]/All Mail,[Gmail]/Trash,[Gmail]/Spam)

[Build id]
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1pre) Gecko/20090621 Shredder/3.0b3pre

[Steps to produce]
(0-1) Folder Properties, Syncronization, [X] Select this folder for offline use
      Execute "Download Now".
     (Gmail IMAP folder. other than [Gmail]/All Mail,[Gmail]/Trash,[Gmail]/Spam)
(0-2) Delete model : Move to Trash
(1) Move 30 mails to other folder, and "compact folder"
    => File size of offilen-store is not reduced
(2) Move back 30 mails to original folder
    => File size of offilen-store increases
(3) Repeat (1) and (2) => File size of offilen-store increases infinitley.

It's probably caused by Gmail IMAP's particularity.

\Deleted flag is kept by [Gmail]/All Mail,[Gmail]/Trash,[Gmail]/Spam only. So "store flag \Deleted" is equivalent to next, if other folder than special folders.
  1. uid XXX store flag \Deleted by Tb.
  2. Before fetch by Tb at step 3, expunge is executed by other client.
  3. fetch by Tb => mail of UID=XXX doesn't exist any more.
     ("deleted mail cont" becomes ZERO due to this?)

Note:
Tb 2.0.0.21 reduces offline-store file size, even when next test(no deletion).
  (1) Rebuild-Index, Download Now
  (2) Repeat (1) several times
  (3) "Compact Folder"
      => creation of nstmp is observed, and offline-store file size is reduced.

Comment 1

[WADA:World Anti-bad-Duping Agency]

I could reproduce problem even with [Gmail]/All Mail...
I tested next.
  1. Move 30 mails from [Gmail]/All Mail to [Gmail]/Trash
  2. Compact [Gmail]/All Mail => offline-store file size is not reduced.
     I couldn't see nstmp.
  3. Move back mails in [Gmail]/Trash to [Gmail]/All Mail, "Download Now"
     => offline-store file size increses. 
  4. Repeat step 1/2/3 => offline-store file size increases infinitely.
It may not be Gmail IMAP particular issue. Possibly DUP of Bug 495862. Setting Dependency.    

I tested with "move to/move back from Trash or [Gmail]/Trash ".
Bug 499634 has relation to the problem?

Comment 2

[WADA:World Anti-bad-Duping Agency]

Tested with move between Inbox and ordinal IMAP folder of Gmail(Gmail Label), and problem is observed. Trash has no relation to this bug(Bug 499634 is irrelevant).

Rough regression window.
(Compact reduced offline-store file size)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081109 Shredder/3.0b1pre
(Compact didn't reduce offline-store file size)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081125 Shredder/3.0b1pre

Note: Bug 466730 is opened on 2008-11-25, and finally fixed around 2009/1/16(Bug 473907 looks to be fixed by the final patch).

Comment 3

[WADA:World Anti-bad-Duping Agency]

Narrowed down regression window.
(Compact reduced offline-store file size)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081121 Shredder/3.0b1pre
(Compact didn't reduce offline-store file size)
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b2pre) Gecko/20081122 Shredder/3.0b1pre

To observe phenomenon easily, next setup is used.
(0) Delete Model: Just mark it as deleted
    mail.server.serverN.offline_download=true(default)
    Inbox : (1 * 1KB mail) + (30 mails, total 200KB in offline-store)
    XXX   : (1 * 1KB mail)
To avoid problems around "expunge command" and "already expunged mail", create offline-store file with Tb rrunk 2009/6/21 build. To avoid problem around "all mails deleted(*0 EXISTS response)", keep at least 1 mail in mail folder.
(1-0)  Restart Tb 2009/6/21 build.
(1-1)  Move 30 mails from Inbox to XXX
       file size of Inbox = 201KB, file size of XXX = 201KB
(1-2)  Move 30 mails from XXX to Inbox
       file size of Inbox = 401KB, file size of XXX = 201KB
(1-3)  Move 30 mails from Inbox to XXX
       file size of Inbox = 401KB, file size of XXX = 401KB
 Note: If above is executed by old builds, deleted and already expunged
       mails won't disappear from thread pane. Because data for these mails
       are held in .msf, "compact operation of offline-store" do nothing.
       It's the reason why 2009/6/21 build is used at this step.
(2) Compact of Inbox => File size of offline-store is not reduced.
(3) Terminate Tb trunk 2009/6/21 build.
(4) Restart Tb trunk 2008/11/22 build, Compact of Inbox
    => File size of offline-store is not reduced.
    Terminate Tb.
(5) Restart Tb trunk 2008/11/21 build, Compact of Inbox, XXX
    => File size of offline-store is reduced.
    Terminate Tb.

I think "Issuing expunge command" and "Compaction of offline-store" should be isolated to support "deleted mails which are expunged by other client" case properly.

Comment 4

[WADA:World Anti-bad-Duping Agency]

Attachment: NSPR log for expungr to Gmail IMAP

Flow is; (IDLE is killed for ease of test)
 (1) 3 mail exist(UID=156,157,158)
 (2) Shift+Delete of UID=156 => uid 156 store flag \Deleted
 (3) Compact => expunge
 (4) Shift+Delete of UID=157 => uid 157 store flag \Deleted
 (5) Compact => expunge

EXPUNGE was issued as expected in this case. But response to expunge from Gmail IMAP was next.
> 00000027 25.24678802 [3900] 4764[3618980]: 3adb800:imap.gmail.com:S-Test-Offline-1/X-01:SendData: 38 expunge 	
> 00000028 25.35819054 [3900] 4764[3618980]: ReadNextLine [stream=3a3c168 nb=15 needmore=0]	
> 00000029 25.35826492 [3900] 4764[3618980]: 3adb800:imap.gmail.com:S-Test-Offline-1/X-01:CreateNewLineFromSocket: 38 OK Success 	

Nothing is returned to expunge, because Shift+Deleted mail is already expunged (same situation as "expunge by other client" before Tb requests expunge).
It's possibly reason why compaction of offline-store is not invoked after 2009/11/22 build.
I guess patch, which sorted out logic around expunge(and compaction) or resolved unwanted/incorrect expunge problem, was landed on 2009/11/22 build.

Comment 5

[WADA:World Anti-bad-Duping Agency]

2009/11/22 => 2008/11/22. Sorry for spam.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Same regression window in next test with Gmail IMAP.
(0) 1*10KB mail in folder XXX. Offline-store file size=10KB.
    ("Move to Trash" model is used)
(1) 2008/11/21 build : OK
    Copy one 10KB mail to XXX => Offline-store file size=20KB
    Open XXX                  => Offline-store file size=30KB (known issue)
    Shift+Delete of one mail  => Offline-store file size=30KB
    Compact                   => Offline-store file size=10KB => OK
(2) 2008/11/22 build : BAD
    Copy one 10KB mail to XXX => Offline-store file size=20KB
    Open XXX                  => Offline-store file size=30KB (known issue)
    Shift+Delete of one mail  => Offline-store file size=30KB
    Compact                   => Offline-store file size=30KB => BAD

Comment 7

[WADA:World Anti-bad-Duping Agency]

Gmail IMAP keeps \Deleted flag for [Gmail]/Trash, and Gmail IMAP returns "* nnn EXPUNGE" to EXPUNGE commmand as ordinaly IMAP server does.
(Two mails exist in [Gmail]/Trash, UID=4704,UID=4705)
(1) Shift+Delete of UID=4704
> imap.gmail.com:S-[Gmail]/Trash:SendData: 16 uid store 4704 +Flags (\Seen \Deleted) 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 1 FETCH (FLAGS (\Seen \Deleted tag-odd) UID 4704) 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: 16 OK Success 
(2) Compact Folder
> imap.gmail.com:S-[Gmail]/Trash:SendData: 17 expunge 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 1 EXPUNGE 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 1 EXISTS 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: 17 OK Success 
> imap.gmail.com:S-[Gmail]/Trash:SendData: 18 getquotaroot "[Gmail]/Trash" 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * QUOTAROOT "[Gmail]/Trash" "" 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * QUOTA "" (STORAGE 182 7517980) 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: 18 OK Success 
> imap.gmail.com:S-[Gmail]/Trash:SendData: 19 UID fetch 4706:* (FLAGS) 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: * 1 FETCH (UID 4705 FLAGS (\Seen tag-odd)) 
> imap.gmail.com:S-[Gmail]/Trash:CreateNewLineFromSocket: 19 OK Success 

Regression window of test of Comment #6 with [Gmail]/Trash was same as usual Gmail IMAP folder(Gmail Label) case.
i.e. "* nnn EXPUNGE" to EXPUNGE is irrelevant to the regression of "Compact Folder doesn't reduce file size of offline-store".

Comment 8

[WADA:World Anti-bad-Duping Agency]

Attachment: Zip of step-by-step .msf files (5 .msf + 1 offline-store)

ZIP file contains next files.
> (A) 2009/07/01 12:20  6,271 X-00.msf-001-10-Active-Mails-Exist-Shutdown
> (B) 2009/07/01 12:22  6,632 X-00.msf-002-Shift-Delete-of-8-Mails-Shutdown
> (C) 2009/07/01 12:25  7,962 X-00.msf-003-Folder-Open-8-Mails-Disappear-Shutdown
> (D) 2009/07/01 12:27  3,798 X-00.msf-004-Folder-Open-Compact-Shutdown-01
> (E) 2009/07/01 12:27  4,039 X-00.msf-005-Folder-Open-Compact-Shutdown-02
> (F) 2009/07/01 12:19 10,240 X-00

[Test scenario]
> (0) Delete Mode: Just mark it as deleted. IDLE is disabled.
>     X-00 is Gmail IMAP folder(Gmail Label). offline-use=yes is set.
> (1) Initial:    X-00 contains 10 mails  (mail-1 to mail-10), Shutdown Tb => (A), (F)
> (2) Restart Tb, Shift+Delete of 8 mails (mail-2 to mail-9),  Shutdown Tb => (B)
> (3) Restart Tb, Open X-00, mail-2 to mail-9 disappears from Thread Pane, Shutdown Tb => (C)
> (4) Restart Tb, Open X-00, Compact Folder, Shutdown Tb => (D)
> (5) Restart Tb, Open X-00, Compact Folder, Shutdown Tb => (E)
> Content/Size/Timestamp of X-00 is not changed from (F) at any step.

[Build ID used in test]
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1pre) Gecko/20090629 Shredder/3.0b3pre

Comment 9

[WADA:World Anti-bad-Duping Agency]

Magnus Melin, can you analyze .msf file content?

Comment 10

[WADA:World Anti-bad-Duping Agency]

Affected by change of nsImapMailFolder.cpp for Bug 420115 on 2008/11/21?


> http://hg.mozilla.org/comm-central/log/b9287cd02d2e/mailnews/imap/src/nsImapMailFolder.cpp
>(snip)
> 6ae64ba03174 2008-11-24 19:55 +0000 Kent James - Bug 461479 - "Integrate generalized Bayes traits into mailnews classifications" [r=Standard8,sr=bienvenu]
> 7a0ccd9061eb 2008-11-24 19:46 +0200 Magnus Melin - Bug 451877: IMAP mark-as-deleted model: while offline - delete doesn't advance to next message. (Should also speed deletion up online quite a bit). r+sr=bienvenu
> e2c807892615 2008-11-21 07:53 -0800 David Bienvenu - don't compact offline store when user does a compact this folder as it interferes w/ the online expunge/compact, r/sr=standard8, 420115
> 74a7415a42a9 2008-11-05 14:37 +0000 Kent James - Bug 419356 - "Allow extensions to add custom filter actions" [r=Neil,sr=bienvenu]
>(snip)

Comment 11

[Magnus Melin [:mkmelin]]

(In reply to comment #9)
> Magnus Melin, can you analyze .msf file content?

Sorry, i never understood much of those :(

Comment 12

[WADA:World Anti-bad-Duping Agency]

Tb's behaviour in next test around the regression window was as follows;
  (Test scenario)
   0. Delete Model : mark as deleted (to see "read cross" mark of deleted mail)
      Folder used : X-00 (file of X-00.msf and X-00)
      Delete X-00 / X-00.msf, restart Tb.
      Do one of nexts to download mails in offline-store:
         - Download Now
         - View each mail
         - Open folder, wait for auto-sync 
      Shutdown Tb, Restart Tb
   1. Shift+Delete mails, and "Compact Folder".
      (never all mails. at least one mail should be kept to avoid other bug)
      ("Expunge is issued or not" is not checked, because test with Gmail IMAP)  
      => Shift+Deleted mails(read cross) is not removed from Thread Pane
   2. Restart Tb, open X-00
      => Shift+Deleted mails(read cross) is removed from Thread Pane
   3. "Compact Folder"

(A) 2008/11/12 to 2008/11/15 build, 2008/11/21 build:
    - Shift+Deleted mails is not removed from Thread Pane until restart of Tb.
    - "Compact Folder" at step 3 reduces offline-store file size.
(B) 2008/11/20 build:
    - Shift+Deleted mails is removed by folder switch only at step 2.
      No need of "restart of Tb".
      (It looks that incomplete patch was landed and removed in next day)
    - "Compact Folder" at step 3 reduces offline-store file size.
--- Start of this bug's problem ---
(C) 2008/11/22 to 2008/11/24 build:
    - Shift+Deleted mails is not removed from Thread Pane until restart of Tb.
    - "Compact Folder" at step 3 dooesn't reduce offline-store file size.
--- End of "not-removed-from-threda-pane-until-restart" ---
(D) 2008/11/22 to 2008/11/24 build:
    - Shift+Deleted mails is removed from Thread Pane
      by first "Compact Folder" at step 1.
    - "Compact Folder" at step 3 doesn't reduce offline-store file size.

"Compact Folders" after 2008/11/22 doesn't invoke Compactor after EXPUNGE completion?

Comment 13

[WADA:World Anti-bad-Duping Agency]

(Correction, sorry for spam)
> --- End of "not-removed-from-threda-pane-until-restart" ---
> (D) 2008/11/22 to 2008/11/24 build:
should be;
  --- End of "not-removed-from-threda-pane-until-restart" ---
  (D) 2008/11/25 build:

Comment 14

[WADA:World Anti-bad-Duping Agency]

Following is a part of change by Bug 420115.
> nsImapMailFolder.cpp
> -  // compact offline store, if folder configured for offline use.
> -  // for now, check aMsgWindow because not having aMsgWindow means
> -  // we're doing a compact at shut-down. TEMPORARY HACK
> -  if (aMsgWindow && mFlags & nsMsgFolderFlags::Offline)
> -    CompactOfflineStore(aMsgWindow);
> ( mapService->Expunge is called just after above )

Following is search result for "CompactOfflineStore" at comm-central.
So no one calls CompactOfflineStore after fix of Bug 420115. 
> CompactOfflineStore
> /mailnews/imap/test/unit/test_compactOfflineStore.js (View Hg log or Hg annotations)
>     * line 69 -- function compactOfflineStore() {
> /mailnews/base/util/nsMsgDBFolder.cpp (View Hg log or Hg annotations)
>     * line 1670 -- nsresult nsMsgDBFolder::CompactOfflineStore(nsIMsgWindow *inWindow, nsIUrlListener *aListener)
> /mailnews/base/util/nsMsgDBFolder.h (View Hg log or Hg annotations)
>     * line 163 -- nsresult CompactOfflineStore(nsIMsgWindow *inWindow, nsIUrlListener *aUrlListener);
> Found 3 matching lines in 3 files

folderCompactor->CompactFolders is called at next.
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#1679
> 1679 nsMsgDBFolder::AutoCompact(nsIMsgWindow *aWindow)
> http://mxr.mozilla.org/comm-central/source/mailnews/base/util/nsMsgDBFolder.cpp#1816
> 1816          if (okToCompact)
> 1817          {
> 1818             nsCOMPtr <nsIAtom> aboutToCompactAtom = do_GetAtom("AboutToCompact");
> 1819             NotifyFolderEvent(aboutToCompactAtom);
> 1820 
> 1821            if ( localExpungedBytes > 0)
> 1822            {
> 1823                nsCOMPtr<nsIMsgFolderCompactor> folderCompactor =
> 1824                  do_CreateInstance(NS_MSGLOCALFOLDERCOMPACTOR_CONTRACTID, &rv);
> 1825                NS_ENSURE_SUCCESS(rv, rv);
> 1826 
> 1827                if (offlineExpungedBytes > 0)
> 1828                  folderCompactor->CompactFolders(folderArray, offlineFolderArray, nsnull, aWindow);
> 1829                else
> 1830                  folderCompactor->CompactFolders(folderArray, nsnull, nsnull, aWindow);
> 1831            }
> 1832            else if (offlineExpungedBytes > 0)
> 1833              CompactAllOfflineStores(nsnull, aWindow, offlineFolderArray);
> 1834          }

So I tried next, and offline-store for [Gmail]/Trash was reduced by Compact Folders.
(1) Disable IDLE command use. Delete Model=Remove it immediately.
    Enable offline-use of [Gmail]/Trash. ([Gmail]/Trash keeps \Deleted flag)
    Move a mail(2KB) to [Gmail]/Trash.
    Restart Tb.
(2) Move 30 mails to [Gmail]/Trash, wait for filling of offline-store file, 
    and delete the 30 mails.
    Because Gmail IMAP keeps \Deleted flag, * NNN EXPUNGED is normally returned
    to EXPUNGE command, as I wrote in Comment #7.
    (due to other bug, at least one mail should remain after delete.)
(3) Click the Gmail IMAP account
(4) File/Compact Folders (not "Comact Folder" of folder context menu)
    => File size of offline-store file for [Gmail/Trash was reduced to 2KB.

Comment 15

[WADA:World Anti-bad-Duping Agency]

Note: "Delete" instead of "Shift+Delete" was used in test of Comment #14.

Comment 16

[WADA:World Anti-bad-Duping Agency]

In test of Comment #14 with 2009/6/30 build using [Gmail]/Trash, first "Compact Folder" of Folder Properties didn't reduce offline-store file size, but second "Compact Folder" Folder Properties reduced offline-store file size.

Problem due to Gmail IMAP can be said as follows?
(a) "Copmact Folders" won't invoke Compactor of offline-store
    unless Tb considers offlineExpungedBytes>0.
(b) offlineExpungedBytes is based on "* NNN EXPUNGED" response to EXPUNGE command.
    Therefore, Compactor of offline-store will never be invoked
    if ordinal Gmail IMAP folder(==Gmail Label).
Next problem also exists?
(c) Shift+Delete affects on localExpungedBytes.

Comment 17

[David :Bienvenu]

(In reply to comment #16)

> (b) offlineExpungedBytes is based on "* NNN EXPUNGED" response to EXPUNGE
> command.
No, expungedBytes is unrelated to NNN EXPUNGED response. It's accumulated as hdrs are deleted from the msg db, based on the offlineMsgSize (for imap folders).

> Next problem also exists?
> (c) Shift+Delete affects on localExpungedBytes.

Offline expunged bytes is just the expunged bytes for folders configured for offline use. Shift delete affects expunged bytes. In other words, there is only one expungedBytes total for all kinds of folders. The local variable "offlineExpungedBytes" just accumulates the expunged bytes for folders configured for offline use.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> > (b) offlineExpungedBytes is based on "* NNN EXPUNGED" response to EXPUNGE
> > command.
> No, expungedBytes is unrelated to NNN EXPUNGED response.
> It's accumulated as hdrs are deleted from the msg db, based on the fflineMsgSize (for imap folders).

Next can be said?
(b-modified)
 hdrs for flagged-\Deleted mail is removed upon NNN EXPUNGED response to EXPUNGE.
 So expungedBytes is not increased if ordinal Gmail IMAP folder(Gmail Label).
 (entry for flagged-\Deleted mail is simply removed upon "fetch 1:* (flag)".)

> In other words, there is only one expungedBytes total for all kinds of folders.

"clear of expungedBytes" is reason why Bug 492344 occurs?

Comment 19

[WADA:World Anti-bad-Duping Agency]

New discovery.
I checked ordinal Gmail IMAP folder case with "Remove it immediately" + Delete(not Shift+Delete) + "File/Compact Folders"(not "Compact" of Folder Properties). "File/Compact Folders" reduced offline-store of ordinal Gmail IMAP folder, and "Done compacting" appeared at status bar.
So (b-modified) was wrong too.

What is difference between "Compact" and "Compact Folders"?

Comment 20

[David :Bienvenu]

(In reply to comment #18)

> Next can be said?
> (b-modified)
>  hdrs for flagged-\Deleted mail is removed upon NNN EXPUNGED response to
> EXPUNGE.

No, that should cause us to delete the header from the db, which makes us adjust the count. If we stop showing you the message in the thread pane, that means we've deleted the header from the db, and adjusted the expunged bytes count

>  So expungedBytes is not increased if ordinal Gmail IMAP folder(Gmail Label).
>  (entry for flagged-\Deleted mail is simply removed upon "fetch 1:* (flag)".)
> 
> > In other words, there is only one expungedBytes total for all kinds of folders.

Right, there's just one count, and for offline imap folders, it's the number of offline bytes expunged.

> 
> "clear of expungedBytes" is reason why Bug 492344 occurs?

that's possible, though we should be maintaining the expungedBytes count when the .msf file is out of date - deleting it outright, on the other hand, would make us lose that count.

Comment 21

[David :Bienvenu]

Plain single folder Compact does not attempt to compact the offline store, unlike the compact folders command, or the compact folders prompt when you open a folder or delete a message and more than XX K bytes will be recovered.

Comment 22

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #21)
> Plain single folder Compact does not attempt to compact the offline store, unlike the compact folders command,

Aha!
Changed by Bug 420115? This bug is INVALID?

Comment 23

[David :Bienvenu]

(In reply to comment #22)
> 
> Aha!
> Changed by Bug 420115? This bug is INVALID?

Yes, that was the change, but given the amount of confusion that caused in this bug, that was probably a bad idea, and I don't think this bug should be invalid.

I'm thinking that compact should not try to repair offline stores (which was what slowed it down a lot), and let auto sync do that.

Comment 24

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #23)
> given the amount of confusion that caused in this bug, (snip)

Amout of confusions by me in this bug is not problem. Contamination of Bug 463359 and Bug 467305 by comments for Tb trunk(next Tb 3) and existence of Bug 495862 is problem which was caused by confusions after Bug 420115. First objective of this bug was to find what caused such wrong comments for Tb trunk(next Tb 3) in Bug 463359 and Bug 467305. It's the reason why I continued to test with "Compact of Folder Properties". I could post comment #19 because I could be aware that you and Wayne Mery said "Compact Folders" instead of "Compact Folder" in Bug 466730 after looking some program codes.

> I'm thinking that compact should not try to repair offline stores (which was
what slowed it down a lot), and let auto sync do that.

Just an idea. Isolate EXPUNGE of server data and COMPACT of locaI offline-store.
 - Change Compact Folders/Compact to Expunge Folders/Expunge if IMAP.
 - New "Compact" in Folder Properties for IMAP. It's similar to Rebuild-Index.
 - New "Compact All Folders" in IMAP Account Properties, if required.

(In reply to comment #21)
> or the compact folders prompt when you open a folder or delete a message and more than XX K bytes will be recovered.

Does it mean "compaction of offline-store of IMAP" is also affected by next?
> mail.prompt_purge_threshhold : true/false
> mail.purge_threshhold        : NNN (in KB, Kilo-Bytes)
> mail.purge.ask               : true/false

Comment 25

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #24)
> I'm thinking that compact should not try to repair offline stores
> (which was what slowed it down a lot), and let auto sync do that.

Another "Just an idea". Change compaction process like next;
  Simple position shifting of data for a mail in active/currently-using
  local mail folder file/offline-store file. Background/independent task. 
    1) Copy an active mail to lower/free position with EXPUNGED bit=ON
    2) Change mail at new position to EXPUNGED bit=Off
    3) Change offset pointer in MsgDB to new position
    4) Change mail at old position to EXPUNGED bit=On
    5) Repeat 1 to 4 for each non-deleted mails
    6) Truncate mail folder file
  1) to 4) is similar to process for delete/detach of attachment.
  For offline-store, next is required additionally.
    0) Set EXPUNGED bit=ON for all non-active offline-store mails
       before start of compactor.
I think change like above can easily resolve next issues.
  - Long locking time of mail data base by Compactor 
  - Bug 495666 (change of mail related data such as tag during compaction)
  - Bug 498814 (interfere by external program especially around delete of
                xxx/xxx.msf and rename of nstmp to xxx)
  - Some bugs listed in Dependency Tree for Bug 498274
Even if problem such as system crash occurs during compaction, there is no need to redo compaction of local mail folder file or offline-store for already shifted mails.
  - Rebuild-Index of local mail folder will ignore already shifted mails
    before crash as designed.
  - There is no need to redo compaction of offline-store,
    because MsgDB doesn't have pointer to non-active mail data in offline-store.

Comment 26

[Wayne Mery (:wsmwk)]

suspect this should be blocking, given that bug 420115 was set so

Comment 27

[WADA:World Anti-bad-Duping Agency]

This bug is never blocker of Tb 3, because "File/Compact Folders" works as designed.
Current issue is;
 a. If implementation change to "Compact doesn't execute compact of offline store"
    is intentional and permanent, it's better to be documented in Release Notes,
    in order to avoid repeatedly opened unwanted bug like this bug or bug 495862.
 b. If change back to "Compact does execute compact of offline store" is planned,
    bug for it should be opened.

Comment 28

[David :Bienvenu]

This should already be fixed - Compact this folder should now also compact the offline store, iirc. I did this when working on some other compaction work having to do with auto sync space constraints.

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #28)
> I did this when working on some other compaction work (snip)

In Bug 482476?
> Bug 482476 Enable Space/Time policy for autosync

Comment 30

[David :Bienvenu]

yes, that's the one.

Comment 31

[WADA:World Anti-bad-Duping Agency]

Gmail IMAP or not was found to be irrelevant. Removing Gmail from bug summary.

Comment 32

[WADA:World Anti-bad-Duping Agency]

Offline-store file size was reduced by "Compact" of folder context menu.
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.3pre) Gecko/20090819 Shredder/3.0b4pre

=> FIXED(by Bug 482476), VERIFIED