765514 - message synchronization does not clean up old messages on imap (StringProperty_pendingRemoval is correctly set, but expungedBytes may wrap or be reset, Compact of offline-store file may fail if offline-store file after Compact exceeds 4GB)

Status: NEW

(MailNews Core :: Backend, defect)

Description

[Corey Hickey]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.4) Gecko/20100101 Firefox/10.0.4 Iceweasel/10.0.4
Build ID: 20120517103601

Steps to reproduce:

Five months ago, I configured thunderbird to "Synchronize the most recent 30 Days", for each of my IMAP accounts.


Actual results:

This behaved as expected, except that, over time, messages synchronized previously were not removed from local disk. For example, my INBOX file is 1.2 GB, and contains messages from the last six months, not the last 30 days.

Using the "Compact" action from the context menu has no effect.


Expected results:

Thunderbird should have periodically removed messages older than 30 days from local disk, in order to reclaim space. I have researched this issue as well as I could, and not found an answer or bug. Other people have had this issue.

https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_3_synchronization_and_reclaiming_local_disk_space

Thanks!

Comment 1

[Hashem Masoud]

(In reply to Corey Hickey from comment #0)
I think you are referring to this: Right-click your folder ➞ Folder properties ➞ Retention policy.

Comment 2

[Corey Hickey]

(In reply to Hashem Masoud from comment #1)
> (In reply to Corey Hickey from comment #0)
> I think you are referring to this: Right-click your folder ➞ Folder
> properties ➞ Retention policy.

Thanks, but this has the following descriptive text:
To recover disk space, old messages can be permanently deleted, both local copies and originals on the remote server.

I don't want them to be removed from the server, I just want them removed locally. In other words, I want them to expire from the local cache.

I rarely access older messages, but I do need to keep them around for those rare occasions.

Thanks,
Corey

Comment 3

[Hashem Masoud]

The closest bug I found asking for this functionality is bug 402730.

Comment 4

[Ludovic Hirlimann [:Usul]]

(In reply to Corey Hickey from comment #0)
> User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.4) Gecko/20100101
> Firefox/10.0.4 Iceweasel/10.0.4
> Build ID: 20120517103601
> 
> Steps to reproduce:
> 
> Five months ago, I configured thunderbird to "Synchronize the most recent 30
> Days", for each of my IMAP accounts.

How did you do that ?

Comment 5

[Corey Hickey]

(In reply to Hashem Masoud from comment #3)
> The closest bug I found asking for this functionality is bug 402730.

Hmm, I hadn't seen that one, but in my opinion it is a separate issue (which is best solved by encrypting the entire disk/filesystem).

I really do like that Thunderbird pre-fetches and caches IMAP messages on local disk. I get a ton of work email and Google's IMAP access is quite slow, so being able to read the messages locally is very nice.

I just think it would be useful for messages older than the specified pre-fetch interval to expire from the local cache, in order to save disk space. I spent quite a while thinking Thunderbird did that, and trying to figure out what I had configured wrong.

Thanks,
Corey

Comment 6

[Corey Hickey]

(In reply to Ludovic Hirlimann [:Usul] from comment #4)
> > Five months ago, I configured thunderbird to "Synchronize the most recent 30
> > Days", for each of my IMAP accounts.
> 
> How did you do that ?

Edit --> Account Settings

For each account:
1. Click "Synchronization and Storage"
2. Check "Keep messages for this account..."
3. Select "Synchronize the most recent..."
4. Choose 30 Days.

Note that this is under Linux, so some locations may be different for you.

-Corey

Comment 7

[Ludovic Hirlimann [:Usul]]

Corey : what's the value of mail.server.default.autosync_offline_stores ?

Comment 8

[Corey Hickey]

(In reply to Ludovic Hirlimann [:Usul] from comment #7)
> Corey : what's the value of mail.server.default.autosync_offline_stores ?

This is set to true (default). I researched the meaning of that setting, and from what I understand, I do want it to be set--I want Thunderbird to continue auto-downloading recent messages, but I want older messages to be auto-removed from the offline store in order to save disk space.

Interestingly, I also came upon an older bug in which the functionality I desire is specifically described as having been implemented:
https://bugzilla.mozilla.org/show_bug.cgi?id=482476#c14
(also see comments 18, 20, and 21)

So, I think one of these three must be true:
1. The code is not functioning properly.
2. I'm doing something wrong.
3. Some other setting is prohibiting the desired behavior.

Thanks,
Corey

Comment 9

[David :Bienvenu]

Corey, have you tried overriding the retention settings for a particular folder, using folder properties, and see if that eventually causes compaction to reduce the offline store size?

Oh, and there was a bug where the last account would not get compacted. That was fixed in 13.0.1, I believe.

Comment 10

[Corey Hickey]

(In reply to David :Bienvenu from comment #9)
> Corey, have you tried overriding the retention settings for a particular
> folder, using folder properties, and see if that eventually causes
> compaction to reduce the offline store size?

Thanks for the response. For the retention policy, I don't want to change that, since it will presumably remove the messages from the server as well. I did try unchecking "Use my account settings", but I still have "Don't delete any messages" selected.

Manual compaction via the context menu as well as restarting Thunderbird has no apparent effect.

I also tried unchecking "Select this folder for offline use" on the Synchronization tab, but that doesn't seem to have any effect either.

> Oh, and there was a bug where the last account would not get compacted. That
> was fixed in 13.0.1, I believe.

Hmm. I have six accounts on each of three different systems, and the same thing is happening to every account.

I do see deleted message get removed when I compact the folder, so I think compaction is happening--it just isn't deleting old messages from the offline store.

Thanks,
Corey

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Corey Hickey from comment #10)
> (In reply to David :Bienvenu from comment #9)
> > Corey, have you tried overriding the retention settings for a particular
> > folder, using folder properties, and see if that eventually causes
> > compaction to reduce the offline store size?
> 
> Thanks for the response. For the retention policy, I don't want to change
> that, since it will presumably remove the messages from the server as well.

in imap, anything you do on the PC affects that server.  If  you want it different, then you want to be using pop

Comment 12

[Magnus Melin [:mkmelin]]

Generally speaking yes, but this case is special since we're only talking about the offline data.

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to Magnus Melin from comment #12)
> Generally speaking yes, but this case is special since we're only talking
> about the offline data.

so should this bug be confirmed?

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to Corey Hickey from comment #8)
> Interestingly, I also came upon an older bug in which the functionality I
> desire is specifically described as having been implemented:
> https://bugzilla.mozilla.org/show_bug.cgi?id=482476#c14

"Removal of mails by days set in Synchronization in patch for bug 482476" was done in nsImapMailFolder::ApplyRetentionSettings().
> http://mxr.mozilla.org/comm-central/source/mailnews/imap/src/nsImapMailFolder.cpp#1232
> 1232 NS_IMETHODIMP nsImapMailFolder::ApplyRetentionSettings()
> Some relevant MessageFlags is defined by;
> http://mxr.mozilla.org/comm-central/source/mailnews/base/public/nsMsgMessageFlags.idl#10
Rough process was:
- If older than days set in Synchronization/Synchoronize most recent,
  set StringProperty of pendingRemoval=1 in msgHdr of mail, and requests update
  of expungedBytes of msgFolder(increase expungdBytes by mail size)
- After a while, expungedBytes is reflected to msgFolder.
- After expungedBytes update, when Compact is invoked,
  Compact removes mail of StringProperty/pendingRemoval=1(set "")
  from offline-store file(don't copy by upon Compact),
  and remove Offline flag from msgHdr.flag data(not held in offline-store).
So, I checked these flag, data, by small addon for test with following procedure.
1. Download all mails(has old date) to Offline-store file by auto-sync,
   with no age limit of Synchronization setting.
2. Change to "Sychronize most recent 1 days".  
   => After a while, StringProperty of pendingRemoval=1 was set for all mails.
   => After a while(not so early, but not so late in my test),
      expungedByets of msgFoler was updated.
3. Manul Compact => Offline flag was removed from mail,
                    Offline-store file size was changed to ZERO.

i.e. The feature worked as desined, as expected in my quick test.

Comment 15

[WADA:World Anti-bad-Duping Agency]

Attachment: Small addon for test named WinBackMyTrash 0.0.4

This addon adds 4 ToolBar button. button3/button4 is for this bug, and object information is written to Error Console.
How to use.
  Add button3 & button4 to Menu Bar, open Error Console.
  Select an IMAP folder of offline-use=on at folder pane.
  Select a mail at tread pane.
  Click button3
  => show extracted msgFolder information.
  => show extracted msgDBHdr information of all mails in selected folder
  => show all properties of msgDBHdr of selected message of selected folder
  button4 is to see properties/function names of mail related objects
  such as gDBview, msgFolder, msgDatabase, mgDBHdr.
  because all properties is listed Error Console log is long.

Bug opener, check StringProperty/pendingRemoval, expungedBytes, Offline flag etc. with small offline-use=on mail folder, please. (Don't test with big mbox, because myaddon for test atempts to write information of all mails in selected folder to Error Console)

Comment 16

[WADA:World Anti-bad-Duping Agency]

Funny phenomenon was observed by "Synchronize most recent 1 Days", with Auto-Compact=Disabled(to avoid unwanted automatic Compact).
- 3 old mails in an IMAP Offline-use=On folder. Toal size=120 KB.
- StringProperty/pendingRemoval=1, flag of Offline=false, is set for each mail.
  expungedBytes of msgFolder is set to 120KB.
- After a while, expungedBytes is increase by 120 KB, and this "Increase of
  expungedBytes by 120 KB" is repeated(perhaps done on each msgPurge interval).

This may cause wrap of expunged Bytes of an IMAP Offline-use=On msgFolder.
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/src/nsDBFolderInfo.cpp#494
> 494 nsDBFolderInfo::ChangeExpungedBytes(int32_t delta)
> 495 {
> 496     return SetExpungedBytes(m_expungedBytes + delta);
> http://mxr.mozilla.org/comm-central/source/mailnews/db/msgdb/public/nsDBFolderInfo.h#74
> 73   uint64_t  m_folderSize;
> 74   int32_t   m_expungedBytes; // sum of size of deleted messages in folder
=> folder size is already changed to 64bits, but expunged bytes is still 32 bits.

If wrap occurs, Compact of this folder may not be executed upon auto-compact invocation.

If "increment of expungedBytes by ApplyRetentionSettings for the Synchronization setting" is counted as threshold of auto-compact invocation, frequesnt auto-compact may occur.

Another concern; 
  Expunge per 10 deletes or Expunge upon each delete, with Move to trash model
I think this is for issuing expunge command and is not for Compact of offline-store.
If expungedBytes is cleared by this Expunge ater delete mail(s), Compact of this folder may not be executed upon auto-compact invocation.

Corey Hickey(bug opener), check transformation of expungedBytes, StringProperty/pendingRemoval, Offline flag etc. in your environment, please.

Comment 17

[Corey Hickey]

I checked this out on 10.0.1, and I never was able to actually get StringProperty_pendingRemoval to be anything but blank (empty string?).

I then upgraded to 17.0.2, and I actually am able to get older messages removed locally by doing a manual compaction (presumably this happens automatically as well). So, tentatively, it looks like this bug has been fixed somewhere in between 10.0.1 and 17.0.2. Later on, I'll test more on a system with 10.0.1 and confirm.

Thanks a bunch for looking into this, in any case.

Comment 18

[Corey Hickey]

Attachment: outputs from button 3

I take back what I said earlier: StringProperty_pendingRemoval was "1" briefly, according to some further testing (attached).

Comment 19

[Corey Hickey]

I meant 10.0.11 where I had written 10.0.1.

Anyway, I can confirm that 10.0.11 has the correct behavior, at least for manual compaction. It may be that the bug was fixed in the 10.0 branch as well, or maybe I was testing it wrong; my ImapMail directory is "merely" 3.6 GB now, though, whereas it used to be > 10 GB.


I just enabled 30-day limits on a couple accounts that had been unlimited before, one of which has an inbox much larger than mail.purge_threshhold_mb. I'll see if the size automatically reduces eventually (I haven't yet been able to research how frequently compaction occurs).

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to Corey Hickey from comment #19)
> my ImapMail directory is "merely" 3.6 GB now, though, whereas it used to be > 10 GB.

It may be reason why Compact was not executed(offline-store file size was not reduced even by manual Compact) by Tb 10 in the past.

Tb has problem of bug Bug 794303, and if "file size after Compact" > "file size limit of 4GB", Compact is not successfull.
> Bug 794303 Compact folder fails if local mail folder size is near 4GB(less than 4GB) or over 4GB,
> even though bug 462665 was changed to FIXED after patch for bug 402392 was landed on Tb 12.0.
Even though offline-store file itself doesn't have 4GB file size limit, Compact of offline-store file also might have problem around 4GB.
I guess that "Compact of offline-store file in Tb 17" works well even when "file size after Compact" is greater than limit by 32bits integer, or that "offline-store file size after Compact" was fortunately less than 4GB when you tested with Tb 17.
It may be that, after offline-store file size was reduced less than 4GB by Tb 17, anything around Compact of offline-store file works well even with Tb 10 in your environment.

Comment 21

[WADA:World Anti-bad-Duping Agency]

Adding some terms in bug summary for ease of understanding problem and search.

Comment 22

[WADA:World Anti-bad-Duping Agency]

If Compact code is shared between BerkleyStore/IMAP/Offline-store-file and BerkleyStore/Local-Mbox-file, and if "nstmp file size after Compact of Offline-stor file" is greater than 4GB, as known by Bug 794303, Compact fails even when "nstmp larger tan 4GB" is normally created, because variable for actual file size check is currently 32bits unsigned integer.

Setting dependency to Bug 794303, for ease of tracking and future analysis.

Comment 23

[Tomislav Parčina]

I'm running 31.1.1 on Fedora Core 20, and I'm experiencing the same problem. If I tourn on the synchronization, mails never get deleted.

We have tried to use this option in our company (Win 7 and FC20), and the same problem is happening on every computer, so we stoped using it.

Please let me know if I need to send more data, or if there is anything else I can do so that this bug gets confirmed.

Best regards.

Comment 24

[Anje]

I would like to add for clarity that when using IMAP and "Synchronize the most recent 30 Days" option.

There is some confusion over what is expected by "Synchronize the most recent 30 Days"

Currently, it seems to mean that 30 days of emails are downloaded and added to whatever was downloaded previously, so gradually, over time, increasing the number of emails in that synchronisd folder.
If this is the intention then it works fine, but it needs some clarity; some helpful description of what this means.

Some people seem to think that it means 'maintain' my synchronised folder to only contain the last 30 days of emails.

What some people seem to expect is that each time it synchronises, the mbox file is compacted to remove 'marked as deleted' mail, then cleared of good emails more than 30days of age, but do not delete the good emails as they must be left on server. Then synchronised to only shows the most recent 30 Days of downloaded emails.

The person at the link below uses the words 'Delete synchronized messages older than 30 days' when they do not actually mean delete because they do not want to delete the emails off the server.  It is more like a request to refresh the folder to only synchronise and show the lastest 30 days of emails. So preventing a build up of emails.

https://support.mozilla.org/en-US/questions/1026895

Comment 25

[Kent James (:rkent)]

I'm linking to bug 789679 since this current bug seems related to the 4GB issue.

Comment 26

[:aceman]

What WADA says would be dependent on bug 894012, however I'm not sure his theory of expungedBytes wrapping around was confirmed yet.

Comment 27

[Tomislav Parčina]

(In reply to Anje from comment #24)
> I would like to add for clarity that when using IMAP and "Synchronize the
> most recent 30 Days" option.
> 
> There is some confusion over what is expected by "Synchronize the most
> recent 30 Days"
> 
> Currently, it seems to mean that 30 days of emails are downloaded and added
> to whatever was downloaded previously, so gradually, over time, increasing
> the number of emails in that synchronisd folder.
> If this is the intention then it works fine, but it needs some clarity; some
> helpful description of what this means.
> 
> Some people seem to think that it means 'maintain' my synchronised folder to
> only contain the last 30 days of emails.
> 
> What some people seem to expect is that each time it synchronises, the mbox
> file is compacted to remove 'marked as deleted' mail, then cleared of good
> emails more than 30days of age, but do not delete the good emails as they
> must be left on server. Then synchronised to only shows the most recent 30
> Days of downloaded emails.
> 
> The person at the link below uses the words 'Delete synchronized messages
> older than 30 days' when they do not actually mean delete because they do
> not want to delete the emails off the server.  It is more like a request to
> refresh the folder to only synchronise and show the lastest 30 days of
> emails. So preventing a build up of emails.
> 
> https://support.mozilla.org/en-US/questions/1026895

Thank you! I agree this is something that needs to be clerified.

I'm not nativ in English, but as fas as I see it second meaning should be right (cleared of good emails more than 30days of age, but do not delete the good emails as they must be left on server).

Comment 28

[Matt]

(In reply to Wayne Mery (:wsmwk) from comment #13)
> (In reply to Magnus Melin from comment #12)
> > Generally speaking yes, but this case is special since we're only talking
> > about the offline data.
> 
> so should this bug be confirmed?

I have come late to the party,  but yes I think it should be confirmed. If nothing else it is a valid use case.  At worst previously implemented functionality is not working, which is what I get from comment 20.

Comment 29

[Tomislav Parčina]

Both bugs, on which this bug depends on, are RESOLVED FIXED. Hopefully someone will start to work on this ticket soon. :)

Comment 30

[:aceman]

Those bugs should help the case of 4GB folders up to comment 22.

So for your issue (which I think Anje has explained properly now) we can either
1. change the wording in the account manager panel
2. change the behaviour to actualy "synchronize" (thus also remove old messages LOCALLY).

Comment 31

[Tomislav Parčina]

(In reply to :aceman from comment #30)
> So for your issue (which I think Anje has explained properly now) we can
> either
> 1. change the wording in the account manager panel
> 2. change the behaviour to actualy "synchronize" (thus also remove old
> messages LOCALLY).

I believe that the second option is what users need. And that is definitely the option that I need.

So, even if 1st option is accepted, please implement the 2nd option as well.

Comment 32

[Matt]

I counsel caution here.  When connectivity goes to god we clear the offline store.  In support often see people needing to undelete those messages they have in their offline store because they have no other access to them.

I suggest that a very cautious approach be made to actually expunging data as soon as the change occurs, otherwise folk who change ISPs with IMAP will have no way to recover their mail when their old account is "closed down"

Comment 33

[torindel]

(In reply to Matt from comment #32)
> I counsel caution here.  When connectivity goes to god we clear the offline
> store.  In support often see people needing to undelete those messages they
> have in their offline store because they have no other access to them.
> 
> I suggest that a very cautious approach be made to actually expunging data
> as soon as the change occurs, otherwise folk who change ISPs with IMAP will
> have no way to recover their mail when their old account is "closed down"

Note: this option doesn't have to (and shouldn't) be default.

If it will be separate from current functionality, all should be fine.

BUG BUMP :]

Comment 34

[Ben Franske]

I'm hoping desperately to see this too. There are some of us with massive IMAP archives/accounts who simply cannot have messages staying in a local cache for an extended amount of time as it absolutely kills hard drive/SSD space. When I was running spinning hard drives in my laptop I would just put bigger ones in but that's less feasible with SSDs when I probably have over 100GB of IMAP messages on servers accessible through Thunderbird. I need to keep profile size under control but do still want to keep a local cache of recent messages for speed and offline work reasons. *Bug bump.

Comment 35

[Wayne Mery (:wsmwk)]

(In reply to Ben Franske from comment #34)
> I'm hoping desperately to see this too. There are some of us with massive
> IMAP archives/accounts who simply cannot have messages staying in a local
> cache for an extended amount of time as it absolutely kills hard drive/SSD
> space. When I was running spinning hard drives in my laptop I would just put
> bigger ones in but that's less feasible with SSDs when I probably have over
> 100GB of IMAP messages on servers accessible through Thunderbird. I need to
> keep profile size under control but do still want to keep a local cache of
> recent messages for speed and offline work reasons. *Bug bump.