267865 - Virtual folders that are subfolders of IMAP folders are deleted after restarting Thunderbird

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Tom Hessman]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.5) Gecko/20041103 Firefox/1.0RC2
Build Identifier: Thunderbird version 0.9 (20041103)

If you create a virtual folder (saved search) and make it a subfolder of an
actual IMAP folder (either through the dialog, or by dragging it into another
folder), it will be gone after restarting Thunderbird.

Reproducible: Always
Steps to Reproduce:
1. Create a new folder on your IMAP server (called "Searches" or similar).
2. Open the new virtual folder dialog (File -> New -> Saved Search).
3. Leave the option for what to make it a subfolder of as the default, which
should be your IMAP server's root.  Also set it to search online.
4. Use any search terms, name it "TestSearch" and click OK to create it.
5. Once it is created, move the virtual folder into the real IMAP folder you
created (Searches).
6. Quit and restart Thunderbird.
Actual Results:  
Once Thunderbird is restarted, the "Searches" folder is empty.  "TestSearch" is
gone.

Expected Results:  
The virtual folder "TestSearch" should still be there, and should still work.

The virtual folders get deleted regardless of whether you create them as a
subfolder of a real folder from the dialog or whether you create the search and
then drag it into a real folder.  The steps above have you drag it into a real
folder after creation to avoid bug 267373.  This bug occurs regardless of
whether or not you experience the issues described in bug 267373 (which I
sometimes see, and sometimes don't).

I'm running the official Thunderbird 0.9 release on Mac OS X (10.3.5).

Setting the severity to "critical" since it causes data loss (the loss of saved
searches, which can potentially be pretty complex and annoying to recreate).

Comment 1

[Mozilla T. Bugs]

I also see this bug in Windows XP (SP1) with the official Thunderbird 0.9 release.

Comment 2

[Tom Hessman]

Yes, I can also confirm it myself on Windows. Setting Platform and OS to All.

Comment 3

[WADA:World Anti-bad-Duping Agency]

Because IMAP folder is placed on IMAP server, trace data of IMAP server and
Thunderbird comunication is usually required for problem analysis by developer.
Tom Hessman and Mozilla T. Bugs,
see http://www.mozilla.org/quality/mailnews/mail-troubleshoot.html#imap
and gather trace data, then attach the trace data(text/plain;) to this bug, please.

Comment 4

[Marcia Knous [:marcia]]

Sarah and I were able to confirm this using today's branch build. Setting TM to
1.0 for consideration, as data loss is involved

Comment 5

[Scott MacGregor]

Attachment: a fix

fix to help with this issue

Comment 6

[Scott MacGregor]

Comment on attachment 166369 [details] [diff] [review]
a fix

I don't know if this will completely fix this bug. I had trouble reproducing
it. The only time I saw strange behavior such as what was described here
occured when I tried to create a virtual folder while an account was selected
in the folder pane. That caused us to end up with a bogus search folder URI
that didn't correspond to a searchable folder.

Comment 7

[Scott MacGregor]

Ths patch has been checked in. Can you guys retest this to see if it full fixes
the symptoms you were seeing?

Comment 8

[Tom Hessman]

At first it seemed to be fixed, but it's actually acting strange and seemingly
inconsistent.  At first, one of the old ones I thought was deleted from one of
the test folders I made showed up, but nothing else did.  Then, after restarting
again, that one was gone, but a bunch showed up in a different test folder. 
Then I tested adding new ones, and in one of the folders they got deleted, but
not in the other (the one with a bunch of old ones).  Then, when I tried making
a new real folder, and dragged a virtual folder in there and restarted
Thunderbird, all the ones from that other folder disappeared, but the ones that
had disappeared from the folder that had only one reappeared.

I decided to try being more straightforward.  After restarting yet again, all
the virtual folders were gone again.  So I made a new one, and dragged it into
the first folder.  After restarting, it was gone from the first folder, but the
second and third folders had their searches back again.  After a third restart,
the first folder had everything back (including the new one), but the other two
folders were empty again.  After another restart, the first folder still had
everything, so I tried one of the searches, and then restarted, after which all
three folders were empty again.

So basically, now it's a mess!  Reopening...

Comment 9

[Scott MacGregor]

please note: old virtual folders created before the patch still have bogus URIS
and won't work.  

Only newly created ones will work if you were having problems before.

Comment 10

[sairuh (rarely reading bugmail)]

(In reply to comment #9)
> please note: old virtual folders created before the patch still have bogus URIS
> and won't work.  

so this would affect 0.9 users who had created virtual folders this way (as
subfolders of an IMAP folder)?

if that's the case, this warrants at least a relnote: "you'll need to recreate
any  saved search folders created under Thunderbird 0.9 as subfolders of IMAP
folders," or something like that.

Comment 11

[Reza Habib]

(In reply to comment #7)
> Ths patch has been checked in. Can you guys retest this to see if it full fixes
> the symptoms you were seeing?

I still get this bug in the version 0.9+ (20041120) build.  Folders are
initially created, but then disappear when TB is closed and reopened.

Comment 12

[Scott MacGregor]

Update: We do propertly create the saved folder and it gets written out to
virtualFolders.dat properly.

The problem is the next time you start up when we read in virtualFolders.dat,
again we parse everything just fine, we find the correct parent folder and we
tell it to add us as a subfolder, calling nsImapMailFolder::AddSubFolder
and I see the folder getting added to mSubFolders.

But then it never shows up in the UI.

And when we quit this 2nd time and write out virtualFolders.dat again, we aren't
finding the folder in mSubFolders anymore so it doesn't get written out. More
details to follow....

Comment 13

[Scott MacGregor]

nsMsgAccountManager::LoadVirtualFolders has the only reference to the imap sub
folder with the virtual folder as a child. As soon as we are done loading
virtual folders, we release the sub folder and it gets destroyed. As a result we
lose mSubFolders and forget about the virtual folders.

Later when the UI comes up, we create the sub folder again through RDF but now
it know longer knows about any saved search folder children. 

Comment 14

[Scott MacGregor]

Further evidence, mParent and mServer on the parent folder are still null after
we get the parent folder from the data source.

It looks like we don't end up creating our folder hierarchy persistently until
the folder data source itself enumerates over the folders to show them in the UI:

nsMsgFolderDataSource::createFolderOpenNode

is the first call I see that actually starts creating nsMsgFolder objects that
stick around and force generation of the sub folders array. So we'd have to call
LoadVirtualFolders at some point after this (which is pretty vague because thsis
gets called once for each top level folder as the xul template builder walks the
datasource). Or we call it inside of GetSubFolders but that'd be pretty expensive.

Comment 15

[Scott MacGregor]

Attachment: the fix...call loadVirtualFolders after our folder hierarchy is established

Comment 16

[Scott MacGregor]

Comment on attachment 166795 [details] [diff] [review]
the fix...call loadVirtualFolders after our folder hierarchy is established

David, there was one weird thing here. AddSubfolders fails for local folder
saved searches because a folder with that name already exists (because the
folder discovery already found the saved search). In this scenario childFolder
is null and we don't generate the spurious ItemAdded notification. 

I also ignored the rv in that case becasue it was throwing a JS exception by
the JS caller of loadVirtualFolders

Comment 17

[Scott MacGregor]

removing the release notes keyword because it no longer applies now that we've
fixed this problem 

Comment 18

[Scott MacGregor]

fixed branch and trunk

Comment 19

[Tom Hessman]

I can confirm that it seems to be fixed correctly now on Mac OS X, 2004-12-02 build.

However, it should be noted that at first, all of my old searches that had
disappeared stayed gone (as comment 9 indicated).  But then, while testing to
see if this was resolved, I created a new virtual folder and dragged it into the
same IMAP folder as a bunch of my old virtual folders, and after restarting
Thunderbird all of my old ones came back, including the ones from the other test
folders!