707446 - Each restart of Tb creates an incremental Inbox-x.msf file for gmail imap account.

Status: NEW

(MailNews Core :: Database, defect)

Description

[alta88]

i noticed panacea.dat had grown to 2M and had references to things like "Documents and Settings", obviously files that no longer existed on an OS that was no longer current.

so i renamed it.  on restart, eventually a script busy dialog appeared, with the Continue prompt, stopped at:
Script: resource:///modules/iteratorUtils.jsm:117

selecting Continue resulted in Tb opening, but with a Gmail Imap account (no stored pwd) containing 3556 inbox folders, in the form INBOX-xxxx, with corresponding references in panacea.dat to each.

upon another restart the folders were gone and only the one Inbox remained under the account.  the references in panacea remained.

Comment 1

[WADA:World Anti-bad-Duping Agency]

You saw some or all phenomena of bug 698321, bug 701688, and bug 520437, because you deleted panacea.dat and Gmail IMAP server access was not done when first restart of Tb after deletion of panacea.dat. See [Test-B] in bug 701688 comment #14.

As you say "3556 inbox folders", many .msf files with suffix(and many .sbd directries with suffix too) looks created in the past(phenomnon of bug 520437).
Because panacea.dat is deleted by you, all subdirectories under Mail, ImapMail, News directory is scanned for existent files for mail folder. Directory scan takes long, and directory scan takes longer if many files are held in a directory, and if many directories are scanned, it takes very long. Further, remote directory scan takes far longer than local directory scan.
If I understand correctly, one of purposes of panacea.dat file is to avoid such directory scan in normal use of Tb.
"Script busy" is perhaps due to this kind of directory scan.

"3556 inbox folders" indicates that internal re-subscribe(or loss of "folder is synchronized with server" status) happened at least 3556 times in the past. Why such many internal re-subscribe did happen in your environment? Do you use Portable Thunderbird on removable media? Do you delete panacea.dat daily?

Anyway, I recommend you to clean up too many files/directries with suffix first, to avoid unexpected or unwanted problems.
(1) Unckeck "[] Keep messages for this account on this computer" at Synchronization&Storage of the Gmail IMAP account, and confirm that no Gmail IMAP folder is set as offline-use=on via Advanced button.
(2) Terminate Tb, delete all .msf files(offline-store files. file with no file extension) and .sbd directories under ...\ImapMail\imap.gmail.com(may be imap.gmail-N.com, imap.googlemail.com, imap.googlemail-N.com).
(3) Delete panacea.dat
(4) Restart Tb, click Inbox of the Gmail account.
(5) Access all Gmail folder to force re-fetch of message header of all mails
    in all Gmail IMAP mail folder. A way for it.
    Enable "Check new messages at start up".
    Check "When getting new messages for this account, always check this folder"
    of all Gmail IMAP folder.
    (mail.server.default.check_all_folders_for_new = true can be used temporary)
    (mail.server.serverN.check_all_folders_for_new = true may be available)
(6) After all message header are fetched, ckeck "[] Keep messages for this
    account on this computer", and uncheck offline-use=on of folders for which
    auto-sync is not mandatory, and wait for auto-sync completion.

Comment 2

[WADA:World Anti-bad-Duping Agency]

Note: Even if above clean up is executed, Junk.msf looks always created internally and Junk-1.msf/Junk-1 looks used, if Junk folder of this account is chosen as junk folder for this account. A workaround of this Junk-1,msf problem is "create Junk under Local Folders" and "select Junk folder of Local Folders as junk folder of this IMAP account".

Comment 3

[alta88]

thanks for the detailed analysis.  i will dupe this bug.

this (In reply to WADA from comment #1)
> "3556 inbox folders" indicates that internal re-subscribe(or loss of "folder
> is synchronized with server" status) happened at least 3556 times in the
> past. Why such many internal re-subscribe did happen in your environment? Do
> you use Portable Thunderbird on removable media? Do you delete panacea.dat
> daily?
> 

this account is solely for testing and Imap in Gmail is turned on only occasionally, usually being turned off in favor of Pop.  but yes, there are thousands of files in the [profD]/ImapMail/imap.gmail.com account folder.  i didn't notice that before.  not a portable Tb nor have i deleted panacea.dat before.

so the bug seems to be the creation, for an incorrect reason i think, of the thousands of Inbox-xxx.msf files.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to alta88 from comment #3)
> so the bug seems to be the creation, for an incorrect reason i think,
> of the thousands of Inbox-xxx.msf files.

Even though creation of the thousands of Inbox-xxx.msf files was due to bug 520437, I believe that happening of phenomenon of bug 520437 due to incorrect or unknown reasons is a bug of Tb(apparently, you didn't do intentional manual unsubscribe/re-subscribe, you didn't do use of copied profile many times, you didn't do deletion of panaca.dat, and so on).
Phenonenon of file/directory with suffix due to existence of bug 520437 is merely an evidence that re-subscribe of IMAP mail folder happened in the past by unknown or incorrect reasons.
alta88(bug opener), why can your this bug be dup of bug 520437?
Do you think that "creation of thousands of Inbox-xxx.msf files due to bug 520437 by unknown reason" itself is never apparent *bug* of a software which is called Thundebird?

Comment 5

[alta88]

in fact, the rebuilding of panacea.dat does not cause the creation of the files.  the rebuild causes the incorrectly created files on disk to show in folderpane.

i just noticed that every time i restart Tb, a new Inbox-x.msf is created in the [profD]/ImapMail/imap.gmail.com account folder.

so that is the real bug.  the gmail imap account is not active in Gmail, is not signed in and rarely is used.  yet each restart creates a new .msf file.

Comment 6

[David :Bienvenu]

Wayne is also seeing this. alta88, do you have access to a source level c++ debugger so that you can perhaps figure out what's going on, with some help from me?

Comment 7

[alta88]

unfortunately no, it would be a start from scratch with c++ debuggers.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Following files are seen in my Gmail IMAP local directory.
  INBOX.msf     2011/12/03  17:15   7,227
  INBOX-1.msf   2011/12/03  18:00   1,356
  INBOX-2.msf   2011/12/03  18:04   1,356
  INBOX-3.msf   2011/12/04  18:22   1,327
  INBOX-4.msf   2011/12/05  10:21   1,327
  INBOX-5.msf   2011/12/05  10:59   1,327
  INBOX-6.msf   2011/12/05  11:19   1,327
  INBOX-7.msf   2011/12/05  12:57   1,327
  INBOX-8.msf   2011/12/05  14:13   1,327
  INBOX-9.msf   2011/12/05  15:34   1,483
After several restarts of Tb 8 today.
  INBOX-10.msf  2011/12/06  12:02   5,135
After last restart of Tb 8, Repair Folder of Inbox, Compact of Inbox, Termination.
  INBOX-10.msf  2011/12/06  12:22   5,836

Because I deleted any inbox.sbd and inbox-N.sbd, next .sbd directry only remains and is currently used by Tb.
  INBOX-10.sbd  2011/12/06  11:54    <DIR>

I used Tb 8 only these days. So this is not problem in nightly builds.
I restarted Tb several times today, and increase of suffix is not observed yet.
It seems for me trouble in Gmail IMAP between 12/03 and 12/06.
I always activate SeaMonkey 2.1 for daily use and same Gmail IMAP account is defined in SeaMonkey. So trouble may be relevant to concurrent access of Mbox from multiple IMAP clients.

alta88, do you still see increase of suffix by each restart of Tb?

Comment 9

[WADA:World Anti-bad-Duping Agency]

alta88, can you check timestamp of thousands of Inbox-N.msf files?

Comment 10

[alta88]

i've deleted the 3K already..

but here's the diagnosis.  i deleted all Inbox* files, enabled Imap in Gmail, restarted, and immediately saw an Inbox.msf file created, while the pwd prompt also came up.  cancelling the prompt means that the Inbox itself is not created.  on restart, another Inbox-1.msf is created, pwd prompt, etc etc IF the account is not signed in.

once the account is succdessfully signed in, *then* an Inbox is created.  on restart(s), no more incremental .msf and one no longer needs to sign in next restart.

so an .msf file should not be created unless the underlying files exists, which for Imap it won't unless login succeeds (if it doesn't already exist).  the file creation order is backwards.

and there you have it.

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to alta88 from comment #10)
> so an .msf file should not be created unless the underlying files exists,
> which for Imap it won't unless login succeeds (if it doesn't already exist).
> the file creation order is backwards.

I suspect this also describes the situation I have seen.

Comment 12

[WADA:World Anti-bad-Duping Agency]

I could easily see phenomenon without password prompt.
  0. Successful access to an IMAP account. Inbox.msf or Inbox-N.msf is used.
     (this step is not mandatory)
  1. Delete Inbox.msf, Inbox-N.msf in local directory for the IMAP account.
     (.msf file for Inbox pointed by panacea.dat doesn't exist any more.)
  2. thunderbird.exe -offline -p "prof_for_test", then terminate Tb
  3. repeat step 2 => suffix of Inbox-N.msf is incremented by each restart.
Confirming.

Inbox-N.msf between 12/3 and 12/5 in my environmet was perhaps by above, because I was testing -offline parameter for bug 698321 and bug 701688.

Comment 13

[alta88]

turns out there's more:

while creating an Inbox will prevent new .msf files on restart immediately after its creation and yesterday multiple restarts were fine, this morning restarts have once again started creating Inbox-x.msf files even with the presence of the Inbox.

my hypothesis (since msgDatabase timestamps are in vogue lately) on a second (or perhaps root) path to this error is that an out of date .msf is detected and a new one is created.  as the account is not logged in, timestamps are not getting updated.  maybe the 'corrupted .msf' meme has all along been mostly timestamps.

WADA this may explain the random timings you're seeing.

anyway, the failure is the lack of file cleanup.

Comment 14

[David :Bienvenu]

timestamps aren't relevant for imap folders/db's.

Comment 15

[alta88]

a final observation - on account creation and signin, a [Gmail].sbd folder is created along with [Gmail].msf.  the .msf file had not been there previously though the .sbd folder was; an .msf was created with a signin.

no restart signins and no Inbox-x.msf files since then.

Comment 16

[:aceman]

I can also see this. I created a test IMAP account with a made-up server name. So TB can never successfully connect to it. This is on linux.
I have currently INBOX-X.msf where X=1 up to 146. At each start of TB, 1 new file is created. The files are sometimes also shown in the left folder pane of TB, but currently I have only seen the numbers 84 up to 146. I don't know why folder 1 - 83 were missing in the list, but they are on the disk. After upgrading to TB9 latest beta (from the previous beta of TB9), all of them vanished from the folder pane, but they are still on the disk.

Comment 17

[WADA:World Anti-bad-Duping Agency]

Bug 131589 was for creation of suffixed .msf/suffixed offline-store file when panacea.dat is corrupted, which will force re-fetch of all mails.

Setting dependency to that bug for ease of tracking.

Comment 18

[Alfred Peters [:infofrommozilla]]

I don't think this bug is limited to gmail. I see it also with several other IMAP accounts. I just deleted another 700 INBOX files.

A little less often I see the error with other folders as described in bug 1287223.

Comment 19

[Alfred Peters [:infofrommozilla]]

(In reply to WADA:World Anti-bad-Duping Agency from comment #12)

I could easily see phenomenon without password prompt.

I can reproduce this with a clean Profile even without offline mode:

1. Create a new profile.
2. Create a new IMAP account.
3. Disable "Check for new messages at startup".
   Disable "Check for new messages every [n] minutes".
4. Select the INBOX an wait till it's synchronized.
5. Select a Folder outside that account (e.g. Local Files)
6. Close TB.

Look into the profile folder: My inbox.msf had a size of ~16kB because some mails were in it.

6. Delete the inbox.msf.
7. Start TB - but do nothing.
8. Close TB.

A new inbox.msf was created. it has a size below 2kB because it's still empty.

Every time you repeat step 7 and 8 a new inbox-N.msf will be created.
I guess TB recognizes the empty inbox file as invalid and therefor creates a new one.

9. Start TB and enter the INBOX

Now the data will be synchronized again and on restart no new inbox-N.msf will be created any more.

Comment 20

[gene smith]

(In reply to Alfred Peters from comment #19)

I did what you did and see it too.

Also, one more thing. If there's a subfolder under inbox, when you finally select Inbox you get a new INBOX-2.sdb directory with all the subfolder under it and the original INBOX.sdb directory is now, I assume, abandoned.