560347 - Saved Searches Being Deleted from virtualFolders.dat (virtual folder is put in scope="list of search target folder URI" with space escaped by %20)

Status: NEW

(Thunderbird :: Folder and Message Lists, defect)

Description

[Happy Chap]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4

Periodically, saved searches are deleted from virtualFolders.dat without any (obvious) action from the user.

Reproducible: Sometimes

Steps to Reproduce:
1. Start Thunderbird
2. Previously generated Saved Search is missing from Folder Pane
3. Specific Saved Search is also no longer in virtualFolders.dat



I have hesitated about reporting this bug because it happens only spasmodically and I cannot determine what is trigger the action. There is NO obvious user action that is generating this (so there is definitely no conscious deleting or editing of the Saved Searches) but it has now happened sufficiently often over a couple of months to indicate that there is a real problem.

Oddly, AFAIK, the Saved Searches disappear in odd (numerically speaking) order. So if you have six Saved Searches number 1-6, then Saved Search 1, 3 or 5 will disappear rather than 2,4 or 6 and this seems(as best I can tell) to happen consistently.

Comment 1

[Ludovic Hirlimann [:Usul]]

does this happens in safe mode? (see https://support.mozillamessaging.com/en-US/kb/Safe+Mode for more information)

Comment 2

[David :Bienvenu]

Can you attach a virtualFolders.dat file from before the saved searches get deleted?

Comment 3

[Happy Chap]

Attachment: virtualFolders.dat backed-up on 8Apr10 with all Saved Searches Contained

Comment 4

[Happy Chap]

Attachment: virtualFolders.dat backuped on 9Apr10 with two Saved Searches now missing

Comment 5

[Happy Chap]

(In reply to comment #2)
> Can you attach a virtualFolders.dat file from before the saved searches get
> deleted?

Hi David,

I've attached two copies of virtualFolders.dat backup on successive nights. The later one is missing two Saved Searches (titled "5. Scouting" and "3. School") which I can confirm were not deleted or edited through the Thunderbird UI.

Thanks, David.

Comment 6

[Happy Chap]

(In reply to comment #1)
> does this happens in safe mode? (see
> https://support.mozillamessaging.com/en-US/kb/Safe+Mode for more information)

Hi Ludovic,

The honest answer is I don't know. It can be several days or even a couple of weeks with no instance of this problem (which is obviously quite a while to run in Safe Mode). If the bug presented more frequently I would try running in Safe Mode without any add-ons.

Comment 7

[David :Bienvenu]

We only try to save virtualFolders.dat when we think it has changed, so that may be why you only sporadically see this issue. My guess is that the issue has to do with all the escapable characters in your saved search names, though all your saved searches seem to have them.

Comment 8

[Happy Chap]

Thanks David.

The virtualFolders.dat had been changed (or at least been 'touch'ed - ie. the modification dates/times had changed) each day from 4th April to 7th April (I have backups of each of those days too), although the Saved Searches weren't lost on any of these 3 changes, so it doesn't appear to be as straight-forward as the escapable characters causing loss each time virtualFolders.dat is updated.

I'm happy to rename the Saved Searches to remove the escapable characters and give it a while to see if the problem reoccurs to give a lead as to where the problem lies.

However, just for info, I've used the same names (with same escapable characters) for Saved Searches under TB2 for a long, long time without any problem at all.

Thanks, David.

Comment 9

[WADA:World Anti-bad-Duping Agency]

"Deleted saved search folder" could be observed by next test with Tb 3.0.4.
1. Creatd virtual folders under an IMAP folder, restart Tb, terminate Tb
2. Open <a_virtual_folder>.msf by SakuRa Editor(write mode open).
3. Restart Tb.
   => <a_virtual_folder> disappeared from Thread pane(All Folders view)
4. Terminate Tb
   => <a_virtual_folder> was removed from virtualFolders.dat.

If write open of <a_virtual_folder>.msf fails, Tb looks to consider "<a_virtual_folder>.msf was deleted". Auto backup softwares, Virus scanner for files, ..., can interfere write open of <a_virtual_folder>.msf by Tb.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> virtualFolders.dat backed-up on 8Apr10 with all Saved Searches Contained

Some entries.
> uri=imap://david@linux/2.%20Work%20todo
> scope=imap://david@linux/#To Do|imap://david@linux/#Next Actions|imap://david@linux/3.%20School
> uri=imap://david@linux/4.%20School%20todo
> scope=imap://david@linux/$$Projects|imap://david@linux/$$Projects/Framing Website|imap://david@linux/$$Projects/Full Governors Agenda|imap://david@linux/$$Projects/Governors Annual Audit|imap://david@linux/$$Projects/New HT Recruitment Pack|imap://david@linux/$$Projects/Streetdancewear|imap://david@linux/$$Projects/Video for our website|imap://david@linux/#To Do|imap://david@linux/#Next Actions|imap://david@linux/3.%20School
> uri=imap://david@linux/6.%20Scouting%20todo
> scope=imap://david@linux/$$Projects|imap://david@linux/$$Projects/Framing Website|imap://david@linux/$$Projects/Full Governors Agenda|imap://david@linux/$$Projects/Governors Annual Audit|imap://david@linux/$$Projects/New HT Recruitment Pack|imap://david@linux/$$Projects/Streetdancewear|imap://david@linux/$$Projects/Video for our website|imap://david@linux/#To Do|imap://david@linux/#Next Actions|imap://david@linux/3.%20School
> uri=imap://david@linux/5.%20Scouting
> uri=imap://david@linux/3.%20School
> uri=imap://david@linux/1.%20Work

imap://david@linux/3.%20School is seen in uri raw(saved search folder name) and scope raw(search target folder, should be real folder).
IFAIK, UI to create saved search doesn't show saved search folders in selection list of search target folders.
When, how did you create saved search folder which has saved search folder in scope? Like next?
1. You had real folder of "3. School", created virtual folder(say XXX) for it.
2. You deleted real folder of "3. School".
3. You renamed virtual folder of XXX to "3. School"
Did you create the saved search folder using Tb 3.0xpre build?

FYI.
A known problem relates to spaces in IMAP folder name: bug 380655. It looks different issue from this bug, because that bug is for issue when multiple space exist in parent's folder name.

Comment 11

[WADA:World Anti-bad-Duping Agency]

Following is an entry in virtualFolders.dat for folder of space in name.
Space is escaped in uri row, but is not escaped in scope row. 
> uri=imap://[e-mail]/VFX/P%20Q%20R
> scope=imap://[e-mail]/VFX/X Y Z|imap://[e-mail]/VFX/A B C

"No escape of space in sope row" is also seen in you virtualFolders.dat.
> |imap://david@linux/$$Projects/Video for our website
> |imap://david@linux/#To Do
> |imap://david@linux/#Next Actions
> |imap://david@linux/3.%20School  <= this is exception

When a search target folder of a search folder is unsubscribed, Tb 3.0.4 deleted it from scope= row of the search folder.
And, when all of search target folder of a search folder is unsubscribed, Tb 3.0.4 deleted the search folder.

Cause of problem may be (a) escaped space in scope, and/or (b) virtual folder as search target folder in scope. It may produce "folder not found" condition in special situation.
> |imap://david@linux/3.%20School

Comment 12

[Happy Chap]

(In reply to comment #10)
> (In reply to comment #3)
> > virtualFolders.dat backed-up on 8Apr10 with all Saved Searches Contained
> 
> Some entries.
> > uri=imap://david@linux/2.%20Work%20todo
> > scope=imap://david@linux/#To Do|imap://david@linux/#Next Actions|imap://david@linux/3.%20School
> > uri=imap://david@linux/4.%20School%20todo
> > scope=imap://david@linux/$$Projects|imap://david@linux/$$Projects/Framing Website|imap://david@linux/$$Projects/Full Governors Agenda|imap://david@linux/$$Projects/Governors Annual Audit|imap://david@linux/$$Projects/New HT Recruitment Pack|imap://david@linux/$$Projects/Streetdancewear|imap://david@linux/$$Projects/Video for our website|imap://david@linux/#To Do|imap://david@linux/#Next Actions|imap://david@linux/3.%20School
> > uri=imap://david@linux/6.%20Scouting%20todo
> > scope=imap://david@linux/$$Projects|imap://david@linux/$$Projects/Framing Website|imap://david@linux/$$Projects/Full Governors Agenda|imap://david@linux/$$Projects/Governors Annual Audit|imap://david@linux/$$Projects/New HT Recruitment Pack|imap://david@linux/$$Projects/Streetdancewear|imap://david@linux/$$Projects/Video for our website|imap://david@linux/#To Do|imap://david@linux/#Next Actions|imap://david@linux/3.%20School
> > uri=imap://david@linux/5.%20Scouting
> > uri=imap://david@linux/3.%20School
> > uri=imap://david@linux/1.%20Work
> 
> imap://david@linux/3.%20School is seen in uri raw(saved search folder name) and
> scope raw(search target folder, should be real folder).
> IFAIK, UI to create saved search doesn't show saved search folders in selection
> list of search target folders.
> When, how did you create saved search folder which has saved search folder in
> scope? Like next?
> 1. You had real folder of "3. School", created virtual folder(say XXX) for it.
> 2. You deleted real folder of "3. School".
> 3. You renamed virtual folder of XXX to "3. School"
> Did you create the saved search folder using Tb 3.0xpre build?
> 
> FYI.
> A known problem relates to spaces in IMAP folder name: bug 380655. It looks
> different issue from this bug, because that bug is for issue when multiple
> space exist in parent's folder name.

Hi,

Yes, I can see what you mean - I hadn't spoked that the scope included other Saved Searches and it also looks very odd that they seem to have appended the same Saved Search folder to each one.

In answer to your question, I create he Saved Searches by searching messages first (Edit->Find->Search Messages) creating my selection criteria using the UI grid and then clicking Save As Search Folder at the bottom.

Referring back to the first point I wouldn't have intentionally included another Saved Search in the scope of a Saved Search (and as you say, it doesn't show this anyway in the UI) so it does look like something screwy is going on there. *And* I have re-created these Saved Searches a few times now because they keep disappearing, so it would be odd if this behaviour is occurring repeatedly.

I've lost the Saved Searches again this evening, but will post a new comment concerning that.

Thanks, David.

Comment 13

[Happy Chap]

Following David's suggestion (Comment 7 above) I removed the escapable characters in the Saved Search names (although they continued to include spaces - I hadn't read Comment 10 above at the time) but last night all Saved Searches disappeared when I started TB so the escapable characters may not be the whole answer.

I re-created the Saved Searches again tonight and a couple of observations (not sure whether they give clues or not);

First Observation:
I created Saved Searches as;
- 1Work
- 2Work Todo
- 3School
- 4School Todo
- 5Scouting
- 6Scouting Todo

(as you can tell, I'm using the initial digit to force a sort order) and the odd numbered Saved Searches (1Work, 3School and 5Scouting) were instantly created in the expected order in the Folder Pane (immediately under the Deleted/Trash folder). However, even numbered Saved Searches (2Work Todo, 4School Todo, 6Scouting Todo) appeared way down the Folder Pane list - so it seemed to be ignoring the initial digit on these.

If I close TB and reopen it, the Saved Searches all appear in the correct/expected order in the Folder Pane, so this behaviour only occurs when initially creating them.

Second Observation:
All these Saved Searches reference use the same basic folders as their scope and the only differences is the Tags that they filter on. So Saved Search '1Work' should be using the same scope as '3School'. However, whilst clicking on '1Work' produces no error, clicking on '3School' brings up a "The current command did not succeed. The mail server responded:Mailbox does not exist, or msut be subscribed to.". However, when I check the scope of the both Saved Searches using the UI (Right Click->Properties->Choose...) only the Inbox folder is selected from the tree for *both* Saved Searches.

However (must use that word less!) if I open virtualFolders.dat, just as WADA highlighted in Comment 10 above, the scope for '3School' actuallly has '1Work' appended to it:

uri=imap://david@linux/3School
scope=imap://david@linux/INBOX|imap://david@linux/1Work
terms=AND (tag,contains,$label2)
searchOnline=true

Now, I didn't intentially do this and agree don't think it can actually be done intentially through the UI so perhaps the root of this problem is actually in bug the how the Saved Searches are being created.

Hopefully this might give some clue.

Thanks again, David.

Comment 14

[Happy Chap]

Is there any relevance to why the Saved Searches refer to some Tags by their user-defined 'friendly' name and others by the Tag's index?

In my virtualFolders.dat, the terms of the Saved Searches sometimes refer to some Tags by their user-defined 'friendly' name (eg. tag,isn't,scouting) and in other cases as (tag,isn't,$label2) when the 2nd label also has a user-defined 'friendly' name.

Comment 15

[David :Bienvenu]

(In reply to comment #14)
> Is there any relevance to why the Saved Searches refer to some Tags by their
> user-defined 'friendly' name and others by the Tag's index?

The first five tags are built-in from the 1.0 days, and use the $labelN names.

Comment 16

[Happy Chap]

Actually, I think I can see how a Saved Search is being included in the scope of another Saved Search and I guess it is only likely to happen when you are creating a lot of Saved Searches all in one go.... when you create a Saved Search it appears to append whatever folder you happen in to have selected at the time in the scope and because (correctly) the Saved Searches UI doesn't show other Saved Searches you don't realise that it's done this.

To recreate this behaviour;

1. Create any Saved Search
2. Select the Saved Search created in step 1 in the Folder Pane so that it is displayed.
3. Without moving off that Saved Search in the Folder Pane, now begin a new Saved Search (Edit->Find->Search Messages)
4. The first drop down ('Search for messages in') correctly won't show the Saved Search highlighted in the Folder Pane and will probably default back to the root of the current account. In any case, change this drop down to 'Inbox'
5. Create some criteria for this new Saved Search
6. Click 'Search' button - and all appears right, your filter criteria is being applied to messages in the 'Inbox'
7. So all is looking good and you click 'Save as Search Folder'
8. In new UI dialogue enter a name for this Saved Search
9. You'll notice that *now* the 'Create as a subfolder of' dropdown does indeed have the Saved Search as teh default value that we have highlighted (back in step 2) but you can change this to 'choose this for parent' of the root account you're in
10. Now the crucial bit, Click 'Choose...' to selected folders to search and it appears totally blank - nothing selected - because of course, correctly, Saved Searches aren't shown. So it has not carried through from step 4 the 'Search for message in' selection. However, as a dumb user you just go ahead and tick 'Inbox' which is what you wanted anyway.
11. Click 'OK'
12. Click 'OK'
13. Close the Search Message dialogue box.

Now although all appears fine, if you look in virtualFolders.dat you will see that the scope for this newly created Saved Search has, appended to the end of it, the Saved Search that you highlighted in step 2 above.

So that's how it gets there when the UI doesn't appear to allow it and maybe as WADA says in Comment 11 above, 'It may produce "folder not found" condition
in special situation.'

To try and test if the root of this whole problem is actually a bug in the creation of the Saved Search, I'm going to manually edit my virtualFolders.dat and remove the extraneous appended Saved Searches in the scopes and see if that stops that Saved Searches disappearing. I'll report back.

Just one final, obvious comment - if the bug identified in step 10 above is fixed that will appear to have solved the problem *but* should the devs at some stage decide to introduce the ability to based a Saved Search on another Saved Search then we obviously haven't found the 'special situation' the WADA talked about where doing so may produce a "fodler not found" event.

Thanks, David.

Comment 17

[David :Bienvenu]

thx, David. We shouldn't be including a saved search in the scope, that's for sure. There may be an issue with the ordering of loading saved searches if a saved search has an other saved search in its scope, but the other saved search hasn't been loaded yet from virtualFolders.dat.

Comment 18

[WADA:World Anti-bad-Duping Agency]

Additional check result.

(A) I could produce %20 of imap: URL in scope row of virtual folder.

1. create real IMAP folder named "X Y Z" and "X%2520Y%2520Z" under "V F X 2".
   - Tb requests create "X%20Y%20Z", and X%20Y%20Z is created.
   - Local file of "X%20Y%20Z.msf" looks to be used for the folder.
   - Somehow file of "X%20Y%20Z" was created and mail data was written to it
     when a mail was copied to folder of "X%20Y%20Z". 
   If virtual folder of "X Y Z" and real folder of "X%2520Y%2520Z"(X%20Y%20Z
   is created by Tb), similar virtualFolders.dat to bug opener's one is
   obtained.

(Note-1) If X%20Y%20Z, Tb requests creation of "X Y Z".
(Note-2) Folder of "X%20Y%20Z" is not accessible if folder of "X Y Z" doesn't exists, because Tb probably issues select "X Y Z". And, rename of X%20Y%20Z fails, because Tb requests rename "X Y Z" "newname".
(Note-3) Next error("^^" instead of "/") was observed while I was repeating creation and rename for testimg of this bug. This phenomenon was reported in two bugs, but reason is not apparent yet. 
> The current command did not succeed.
> The mail server responded:Unknown Mailbox: V F X2^^X Y Z (Failure)
It looks wrong folder name due to next garbage in .msf, or garbage in .msf generated by above error.
> <(80=1)(81=V F X2^X%20Y%20Z)(82=0)(83=)(87=1272426483)(85=12)(86
>    ={"threadCol":{"visible":true,"ordinal":"1"},"flaggedCol":{"visible":true,\

AFAIR, these are known phenomenon in other bug(s) which is for % related issue.
 
2. create virtual folder "A B C" under IMAP account.
(virtaulFolders.dat) 
> uri=imap://[e-mail]/V F X 2/A%20B%20C
> scope=imap://[e-mail]/V F X 2/X Y Z|imap://[e-mail]/V F X 2/X%20Y%20Z
(content in "V F X 2.sbd\A B C.msf")
> <(80=1)(83=A B C)(84=24)(85=ALL)(86
>     =imap://[e-mail]/V F X 2/X Y Z|imap://yatter.\
> [e-mail]/V F X 2/X%20Y%20Z)(87=0)(8C=3)(88=)(89

Question again.
When(with which build) did you create the virtual folders initially?
Next may be cause of %20 in scope.
- Tb3.0xpre had problem of "can create virtual folder as subfolder of a virtual
  folder". This bug is already resolved.
- Issues around escaping/unescaping of internal URL especially when IMAP.
  Many of them are resolved but some are still remain.

(B) If parent folder is renamed, virual folders of IMAP, which refere to subfolders of the parent only as search target folders, is deleted.

3. Rename "V F X 2" to "V F X2".
   => following virtual folders is deleted.
   If local virtual folder, rename of parent is reflected to virtual folders
   under the parent normally.
> uri=imap://[e-mail]/V F X 2/A%20B%20C
> scope=imap://[e-mail]/V F X 2/X Y Z|imap://[e-mail]/V F X 2/X%20Y%20Z

Because it seems that "no escape of space of imap: URL format path in scope=" is spec of Tb, "%20" in scope= may produce problems around escaping/unescaping of internal imap: url, and it may produce condition of "parent folder not found" or "serch target folder not found".
As seen in bug 503735, problem of "account is considered root-level folder" really exists.

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #16)
> 7. So all is looking good and you click 'Save as Search Folder'
> 8. In new UI dialogue enter a name for this Saved Search
> 9. You'll notice that *now* the 'Create as a subfolder of' dropdown does
> indeed have the Saved Search as the default value that we have highlighted
> (back in step 2) (snip)

If you keep the existent Saved Search folder at "Create as a subfolder of:", you can create the new Saved Search folder as a sub folder of existent Saved Search folder. It's already known bug 504062 which I wrogly thought already fixed.

Comment 20

[WADA:World Anti-bad-Duping Agency]

After creation of Saved Search folder of "A B C" as subfolder of existent Saved Search folder of "B C D" by utilizing bug 504062, without selection of search target folders via UI, I could observe next in virtualFolders.dat.
> uri=imap://[e-mail]/V F X2/B%20C%20D/A%20B%20C
> scope=imap://[e-mail]/V F X2/B%20C%20D
Folder pre-set in "Create as a subfolder of:" looks to be pre-selected as search target folder, even if the folder is Saved Search folder. 

When "A B C" is renamed to "C D E", virtualFolders.dat was changed to next. 
> uri=imap://[e-mail]/V F X2/B%20C%20D/C%20D%20E
> scope=imap://[e-mail]/V F X2/B%20C%20D
And, when real folders of "/V F X2/P Q R" and "/V F X2/X Y Z" are selected at UI of search target folder of Folder Properties, next entry was created. 
> uri=imap://[e-mail]/V F X2/B%20C%20D/C%20D%20E
> scope=imap://[e-mail]/V F X2/P Q R|imap://[e-mail]/V F X2/X Y Z|imap://[e-mail]/V F X2/B%20C%20D
It seems next;
  Because Saved Search folder of "/V F X2/B%20C%20D" doesn't appear in
  selection list of search target folder, the saved search folder is not
  considered "not selected".
I think change like next is safer than current, if it wont't affect implementation of "Smart Folders" or others like View.
  Remove, if not checked, and add, if checked
  => Put selected folders only in scope= raw of virtualFolders.dat and .msf.

Setting dependency to bug 504062, for ease of tracking.

Comment 21

[WADA:World Anti-bad-Duping Agency]

Key steps.
(1) Click a Saved Search folder at folder pane
(2) (2-A) Edit/Find/Search Messages,
    (2-B) execute search, and Save as Search Folder,
    (2-C) Saved Search folder at (1) is pre-set in "Create as a subfolder of:".
    => The Saved Search folder at (1) is put in scope= of virtualFolders.dat
       and is saved im .msf.
       It's not removed by search target folder selection.

If active folder at (1) is not Saved Search folder, folder set at (2-C) is real folder, then problem wont' occur.
If step (2) is "Search" of folder context menu of Saved Search folder, search is impossible because "Search" doesn't appear in menu, so (2-B) can't happen.
If step (2) is "Search" of folder context menu of other real folder than (1) with right click only, the real folder is set as pre-set folder in "Create as a subfolder of:" at step (2-C), because internally selected/active folder is changed to right-clicked folder even though highlight'ed folder is still folder selected at step (1) (known issue. mismatch between internally selected folder and folder highlighted at folder pane when right click).

To remove virtual folder incorrectly put in scope= from scope= row of virtualFolders.dat;
If path for virtual folder incorrectly put in scope= has escaped string, create virtual folder of the escaped string via File/New/Saved Search, and delete it.
  Create virtual folder of "A%20B", then delete "A%20B".
  Original virtual folder of "A B" itself is not affected.
If path for virtual folder incorrectly put in scope= doesn't have escaped string, delete of the virtual folder and re-creation of the virtual folder is required.

Comment 22

[Happy Chap]

Just to feedback, I did manually edit virtualFolders.dat to remove th erroneous Saved Search folders in the scope of others and having been running this new virtualFolders.dat for almost two weeks now without the problem reoccuring. 

Two weeks *might* be a little short to be conclusive but it does appear to support that this was the problem.

David.