694858 - local copy of IMAP mail destroyed without notice, because required actions to continue access to the local copy was not done even after IMAP account ceased at server

Status: RESOLVED DUPLICATE of bug 312619

(MailNews Core :: Networking: IMAP, defect)

Description

[Pierre Albarède]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Build ID: 20110928134238

Steps to reproduce:

Hello,

for years, I have been using Thunderbird, keeping it up to date regularly (Mac OS X).  I have one IMAP account, configured to keep local copies for offline use.  My subscription to the IMAP service ceased in March 2011.  I used local copies offline for months afterwards, at least until May, without trouble.


Actual results:

Now, I come back to check those old messages and they are all gone.


Expected results:

Fortunately, I do have an incremental backup of the IMAP folder with Time Machine. 
 
It occurs that until July 2, size was 290 MB. 
Folder was modified on July 12, then size was 78 MB on July 17.
Folder was modified on July 24 at 11:40, then size was 258 kB on July 26, since then increasing slowly to 336 kB today.

There are two possibilities :
either I am fool enough to have trashed everything (I confess : I made some cleaning),
either there was a problem with Thunderbird.

I am ready to make whatever tests could resolve that matter, based on Time Machine records.

Probably, if nobody reports similarly, I am the fool one.

Cheers.

Comment 1

[Pierre Albarède]

Cleaning was on July 12.
I do not know what occured on July 24 but I certainly did not take all the steps necessary to completely destroy all messages from within user interface.

Comment 2

[Pierre Albarède]

secure log

Jul 24 11:31:33 macbook-pro com.apple.SecurityServer[23]: Session 0x400569 dead
Jul 24 11:31:33 macbook-pro com.apple.SecurityServer[23]: Killing auth hosts
Jul 24 11:31:33 macbook-pro com.apple.SecurityServer[23]: Session 0x400569 destroyed
Jul 24 11:31:34 macbook-pro com.apple.SecurityServer[23]: Session 0x43214a created
Jul 24 11:31:34 macbook-pro com.apple.SecurityServer[23]: Session 0x43214a attributes 0x30
Jul 24 11:37:22 macbook-pro com.apple.SecurityServer[23]: Session 0x43214a dead
Jul 24 11:37:22 macbook-pro com.apple.SecurityServer[23]: Killing auth hosts
Jul 24 11:37:22 macbook-pro com.apple.SecurityServer[23]: Session 0x43214a destroyed
Jul 24 11:38:42 macbook-pro com.apple.SecurityServer[22]: Session 0x5fbff962 created
Jul 24 11:38:42 macbook-pro com.apple.SecurityServer[22]: Entering service
Jul 24 11:39:02 macbook-pro com.apple.SecurityServer[22]: Succeeded authorizing right 'config.modify.com.apple.CoreRAID.admin' by client '/System/Library/PrivateFrameworks/CoreRAID.framework/Versions/A/Resources/CoreRAIDServer' for authorization created by '/System/Library/PrivateFrameworks/CoreRAID.framework/Versions/A/Resources/CoreRAIDServer'
Jul 24 11:39:10 macbook-pro com.apple.SecurityServer[22]: Session 0x20bd47 created
Jul 24 11:39:10 macbook-pro com.apple.SecurityServer[22]: Session 0x20bd47 attributes 0x30
Jul 24 11:39:12 macbook-pro loginwindow[41]: Login Window Started Security Agent
Jul 24 11:39:17 macbook-pro SecurityAgent[199]: Showing Login Window
Jul 24 11:39:31 macbook-pro SecurityAgent[199]: User info context values set for alba
Jul 24 11:39:32 macbook-pro SecurityAgent[199]: Login Window Showing Progress
Jul 24 11:39:33 macbook-pro SecurityAgent[199]: Login Window done
Jul 24 11:39:33 macbook-pro com.apple.SecurityServer[22]: Succeeded authorizing right 'system.login.console' by client '/System/Library/CoreServices/loginwindow.app' for authorization created by '/System/Library/CoreServices/loginwindow.app'
Jul 24 11:39:33 macbook-pro loginwindow[41]: Login Window - Returned from Security Agent
Jul 24 11:39:33 macbook-pro loginwindow[41]: MDS Error: unable to create user DBs in /var/folders/kD/kDLiV3Z42RWMZU+8ZP5RSU+++TI/-Caches-//mds
Jul 24 11:39:33: --- last message repeated 2 times ---
Jul 24 11:39:33 macbook-pro com.apple.SecurityServer[22]: Succeeded authorizing right 'system.login.done' by client '/System/Library/CoreServices/loginwindow.app' for authorization created by '/System/Library/CoreServices/loginwindow.app'
Jul 24 11:39:43 macbook-pro /System/Library/CoreServices/CCacheServer.app/Contents/MacOS/CCacheServer[236]: Starting up.
Jul 24 11:49:43 macbook-pro /System/Library/CoreServices/CCacheServer.app/Contents/MacOS/CCacheServer[236]: No valid tickets, timing out
Jul 24 12:11:20 macbook-pro loginwindow[41]: in pam_sm_authenticate(): Failed to determine Kerberos principal name.
Jul 24 12:11:21 macbook-pro com.apple.SecurityServer[22]: Succeeded authorizing right 'com.bombich.ccc.PrivilegedTask' by client '/Applications/CCC/3.2.1 b1/Carbon Copy Cloner.app/Contents/MacOS/ccc_helper.app' for authorization created by '/Applications/CCC/3.2.1 b1/Carbon Copy Cloner.app/Contents/MacOS/ccc_helper.app'


CCC is another backup software.  (Yes I am paranoid.)

Comment 3

[Ludovic Hirlimann [:Usul]]

If you log imap as described at https://wiki.mozilla.org/MailNews:Logging , do you see activity for that account (imap and ImapAutoSync) ?

Comment 4

[Pierre Albarède]

Hello,
there could be no activity because the IMAP service stopped at end of March.

Comment 5

[Ludovic Hirlimann [:Usul]]

Does repairing the folder helps ? right click -> properties -> rebuild ?

Comment 6

[Pierre Albarède]

I tried but it does not help.  Actually, recovering the data is not my primary interest.  I would like to know why they have disappeared in order to avoid another occurence.  Thanks.

Comment 7

[Pierre Albarède]

I have copies of the IMAP folder and its index (.msf) before and after the data (INBOX, Sent) loss.

BEFORE (Time machine record on 2011 07 15)

macbook-pro% ls -lR /Users/alba/Desktop/20110715 
total 8
drwxrwx---  14 alba  staff   476 Jul 12 18:51 imap.cea.fr
-rw-rwx---   1 alba  staff  1143 Mar 28  2011 imap.cea.fr.msf

/Users/alba/Desktop/20110715/imap.cea.fr:
total 152488
-rw-------@ 1 alba  staff      1749 Jul 12 18:51 Drafts
-rw-r--r--@ 1 alba  staff      4591 Jul 12 21:35 Drafts.msf
-rw-------@ 1 alba  staff  44292772 Jul 12 18:51 INBOX
-rw-r-xr--  1 alba  staff    142427 Jul 12 21:35 INBOX.msf
-rw-------@ 1 alba  staff         0 Jul 12 18:51 Junk
-rw-r-xr--  1 alba  staff      6041 Jul 12 21:35 Junk.msf
-rw-------@ 1 alba  staff  33526166 Jul 12 18:51 Sent
-rw-r--r--@ 1 alba  staff     64064 Jul 12 21:35 Sent.msf
-rw-------@ 1 alba  staff         0 Jul 12 18:51 Trash
-rw-r--r--@ 1 alba  staff     12421 Jul 12 21:35 Trash.msf
drwxrwx---  3 alba  staff       102 Dec 17  2010 Trash.sbd
-rw-r--r--@ 1 alba  staff       207 Jan 16  2011 msgFilterRules.dat

[...]

macbook-pro% ls -lR /Users/alba/Desktop/20110726 
total 8
drwxrwx---  12 alba  staff   408 Jul 24 11:40 imap.cea.fr
-rw-rwx---   1 alba  staff  1219 Jul 21 21:10 imap.cea.fr.msf

/Users/alba/Desktop/20110726/imap.cea.fr:
total 504
-rw-r--r--@ 1 alba  staff    4591 Jul 12 21:35 Drafts.msf
-rw-r--r--@ 1 alba  staff    1620 Jul 19 22:38 INBOX-1.msf
-rw-r--r--@ 1 alba  staff    2019 Jul 22 13:48 INBOX-2.msf
-rw-r--r--@ 1 alba  staff    1620 Jul 24 13:57 INBOX-3.msf
-rw-r-xr--  1 alba  staff  142427 Jul 12 21:35 INBOX.msf
-rw-r-xr--  1 alba  staff    6041 Jul 12 21:35 Junk.msf
-rw-r--r--@ 1 alba  staff   64064 Jul 12 21:35 Sent.msf
-rw-r--r--@ 1 alba  staff   12421 Jul 12 21:35 Trash.msf
drwxrwx---  3 alba  staff     102 Dec 17  2010 Trash.sbd
-rw-r--r--@ 1 alba  staff     207 Jan 16  2011 msgFilterRules.dat

[...]

How to determine from this the date of data loss ?

Assuming that the last modification of imap.cea.fr was data loss
(could it be something else ?), I inferred
Jul 24 11:40 


However indexed was rebuilt
-rw-r--r--@ 1 alba  staff    1620 Jul 19 22:38 INBOX-1.msf
probably because INBOX was already lost at this time.

Thus the data loss may have occurred before Jul 19 22:38.
But why is not there Sent-1.msf ?

Your guess ?

Knowing the date, I can look for concurrent processes in logs.

Comment 8

[David :Bienvenu]

An IMAP protocol log of a session where this happens would be helpful - https://wiki.mozilla.org/MailNews:Logging

Comment 9

[Wayne Mery (:wsmwk)]

Pierre ??

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to Pierre Albarède from comment #0)
> My subscription to the IMAP service ceased in March 2011.
> I used local copies offline for months afterwards, at least until May, without trouble.
> Actual results:
> Now, I come back to check those old messages and they are all gone.

Once sync'ed status of IMAP mail folder is lost, it's impossible to use IMAP offline-store file, because IMAP always relies on IMAP server.
So, if login to IMAP server and mbox access via IMAP is killed at server, "Work Offline" mode of Tb is always needed to access mail data which is already downloaded to IMAP offline-store file by Tb, in order not to loose sync'ed status.

> -rw-r-xr--  1 alba  staff  142427 Jul 12 21:35 INBOX.msf
> -rw-r--r--@ 1 alba  staff    1620 Jul 19 22:38 INBOX-1.msf
> -rw-r--r--@ 1 alba  staff    2019 Jul 22 13:48 INBOX-2.msf
> -rw-r--r--@ 1 alba  staff    1620 Jul 24 13:57 INBOX-3.msf

This indicates that "sync'ed status is lost at Jul 19 22:38" and similar thing to "unsubscribe and try to re-subscribe" happend then phenomenon of bug 520437 occurred and INBOX-1.msf is used for Inbox folder.

Did you start Tb with Offline Mode(thunderbird.exe -offline)?
Or you restarted Tb with Work Online mode and tried to access IMAp server even though IMAP account is killed at server?   

Once sync'ed status is lost, similar phenomenon to bug 698321 can happen.
After that, because IMAP server status is always same as offline mode due to kill of IMAP account at server, phenomenon bug 707446 can happen upon any restart of Tb.

Recovery procedure.

Fortunately, you looks to have backup of file of Inbox(offline-store file),
> /Users/alba/Desktop/20110715/imap.cea.fr:
> -rw-------@ 1 alba  staff  44292772 Jul 12 18:51 INBOX
and file size is less than 2GB(only 44MB).
So, if you copy this file to .../Mail/Local Folders/Inbox-of-IMAP, you can see mails as mails in local mail folder named Inbox-of-IMAP under "Local Folders".
However, never copy mail in Inbox-of-IMAP(it was originally IMAP ofline-store flle) to IMAP server(i.e. never upload to IMAP Mbox).
If you do upload, you will see bug 426651, bug 667288, bug 697635, and bug 708941.

If you need tag(flag) status, Unread status, Replied status etc., you can access old IMAP data by next procedure, if and only if you have full&helthy backup of Tb's profile.
(1) Call currently used profile "P1".
    Copy P1 to P2.
    (Copy profile directory, add copied profile directory as directory for P2)
(2) Use P2 as daily use profile of Tb.
Because other IMAP accounts are active, all mail data can be re-downloaded from server even if something wrong happen.
If you need, clean up garbages of xxx-N.msf, xxx-N, xxx-N.sbd manually.
(3) Restore full & helthy backup of P1 to profile directory for P1.
Note: Absolutely path which saved in panaca.dat is correct, because restore to original location.
(4) Restart additional Tb using P1, with offline mode always.
    thunderbird.exe -offline -p "P1"
    thunderbird.exe -no-remote -offline -p "P1"
   (Tb can run concurrently by -no-remote, but be careful because very confusing)

Comment 11

[Pierre Albarède]

Hello

WADA wrote

> you restarted Tb with Work Online mode and tried to access IMAp server even though IMAP account is killed at server?   

That is likely and it should not be harmful.

>"Work Offline" mode of Tb is always needed to access mail data

> local copy of IMAP mail destroyed without notice, because required actions to continue access to the local copy was not done even after IMAP account ceased at server

Shall I understand that local copy of IMAP mail was destroyed when trying to work online with broken IMAP service ?  That would be a terrible bug, also easy to reproduce.


>never upload to IMAP Mbox

I tried and indeed it failed.  I understand that IMAP folder must be touched only by IMAP server.

However, I could import in local mail folders.

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to Pierre Albarède from comment #11)
> >"Work Offline" mode of Tb is always needed to access mail data
> 
> > local copy of IMAP mail destroyed without notice, because required actions to continue access to the local copy was not done even after IMAP account ceased at server
> 
> Shall I understand that local copy of IMAP mail was destroyed when trying to
> work online with broken IMAP service ?  

yes. because (unfortunately in your case) the local copy is intended to be a mirror of what is on the server

Comment 13

[Wayne Mery (:wsmwk)]

bug 312619 would have helped you