754799 - saved search / virtual folders don't remember folders to search for 3rd+ level of *local* folders (starting with thunderbird 12)

Status: RESOLVED WORKSFORME

(Thunderbird :: Search, defect)

Description

[u442193]

User Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0
Build ID: 20120420145725

Steps to reproduce:

I generated a saved search / virtual folder.


Actual results:

After restarting Thunderbird 12 saved search / virtual folder does not remember all marked folders.


Expected results:

I assume it should remember all folders marked for search.

Comment 1

[u442193]

TB12 remembers 1st and 2nd level folder hierarchy properly. Starting from 3rd level it does not.

Comment 2

[Ludovic Hirlimann [:Usul]]

Anything in tools -> Error console ?

Comment 3

[David :Bienvenu]

I'm skeptical that there is something special about 3rd level folders in general. Are these imap folders or local folders? Can anyone reproduce this?

Comment 4

[u442193]

@comment from David

I can 100# reproduce this on my Windows XPx32 machine. Mark all folders, then restart. 1st and 2nd level still marked from 3rd level on unmarked. I can supply a screen shot if you want.

This phenomenon comes in conjunction with a repeated generation of MSF-files. Everytime I restart TB, I mark all folders for a saved search. Then select another folder and then the saved search again, TB will generate new summary (msf) files for some folders. I will try to find out which folders.

Regards,

Thomas

Comment 5

[u442193]

(In reply to Ludovic Hirlimann [:Usul] from comment #2)
> Anything in tools -> Error console ?

I get 3 errors, 4 warning and many messages. What should I look for?

Comment 6

[u442193]

(In reply to David :Bienvenu from comment #3)
> I'm skeptical that there is something special about 3rd level folders in
> general. Are these imap folders or local folders? Can anyone reproduce this?

These are local folders.

Comment 7

[u442193]

(In reply to David :Bienvenu from comment #3)
> I'm skeptical that there is something special about 3rd level folders in
> general. Are these imap folders or local folders? Can anyone reproduce this?

I can reproduce on another machine with Windows XPx32.

Comment 8

[David :Bienvenu]

(In reply to bugzilla from comment #4)

> 
> This phenomenon comes in conjunction with a repeated generation of
> MSF-files. Everytime I restart TB, I mark all folders for a saved search.
> Then select another folder and then the saved search again, TB will generate
> new summary (msf) files for some folders. I will try to find out which
> folders.
Yes, I suspect the problems are related. Since I don't see the .msf file issue, I don't see the saved search issue either. What exactly are the folder names and full paths to the folders that have the issues with the .msf files? You can e-mail me the info if you don't want the full paths in the bug...

- David

Comment 9

[Ludovic Hirlimann [:Usul]]

^^^ [e-mail]  ?

Comment 10

[u442193]

Issue is still pending. I migrated my complete Thunderbird installation to Windows7x64 now running TB 17.0.2

In the meantime if have found out that this currently only happens in folders with e-mails that come from Sweden. I am not an expert, but could this be some kind of language coding / date issue?

Regards,

Thomas

Comment 11

[u442193]

(In reply to Ludovic Hirlimann [:Usul] from comment #9)
> ^^^ [e-mail]  ?

Don't you like this? My way of making it easier to keep track of things!

Comment 12

[u442193]

Hello Programmers,

may be I have found the reason for this issue. Unfortunately I cannot fix it.

I have taken the folder and it subfolders that have been indexed every time I open the virtual filter and renamed it from Software to AAA! What happens now is that TB is indexing the last folders e.g. ZZZ every time I open the virtual folder. So to me this looks like TB only remembers a certain number of folders and forgets the others because they are of of range?

Hope this help.

Comment 13

[:aceman]

How many folders do you have? On the folder that gets reindexed, try rightclicking it -> properties -> repair folder.

Comment 14

[u442193]

(In reply to :aceman from comment #13)
> How many folders do you have? On the folder that gets reindexed, try
> rightclicking it -> properties -> repair folder.

I have a little more than 512! folders. Fixing will not help, because as described above, if I put the same folder that is now the last in the TB sort order, together with all its subfolders alphabetically as first folder, TB will not try to reindex this folder any more. It will then try to reindex the "newly" last folder of the sort order.

It will always be that last folders in the alphabetical sort order of TB.

Comment 15

[WADA:World Anti-bad-Duping Agency]

> Bug summary : 
>   saved search / virtual folders don't remember folders to search for
>   3rd+ level of folders

Which option do you use at Server Settings/Advanced?
  "Show only subscribed folder" is Checked, or Uncheced

If Unchecked, phenomenon may be one like next.
Assume Inbox, Inbox/A, Inbox/A/B, Inbox/A/B/C, Inbox/A/B/C/D, ...
(a) Account is collapsed upon restart
(b) Tb doesn't request LSUB when "Show only subscribed folder" is Unchecked.
    Instead, try to use only LIST for all folders.
    Request upon restart is for only two level below - LIST %/%, LIST Inbox/%/%.
    Upon expansion of Mbox, LIST Mbox/%/% is reuested.
    And, when collased Mbox is double-clicked, LIST Mbox/% is also requested.
(c) If server has problem like bug 710507(nothing is returned to LIST Mbox/%/%,
    returns subfolder names only for LIST Mbox/%), lower level folders can not
    be know once folder structure is lost, until a folder is double-click'ed.
(d) By (b), when Inbox is collapsed upon restart,
    if folder structure is somehow lost upon restart,
    Inbox, Inbox/A, Inbox/A/B is known, but Inbox/A/B/C and below is unknown.
    Problem like (c) easily make it worse.
(e) "Re-indexing of .msf upon resatrt" in your case may be a phenomenon of
    "folder structure is somehow lost upon restart" what I called in (d).
(f) When a folder poined in virtualFolders.dat is not found, Tb removes the
    missing folder from search target folder list in virtualFolders.dat.
(g) If "Show only subscribed folder" is Checked, LSUB is used upon restart of Tb.
    Because only existent and subscribed folder can be selected upon
    search target folder request for a Virtaul Folder(Search Folder),
    folder used by a Virtual Folder is usually contained in LSUB response.
    So, no problem occurs when "Show only subscribed folder" is Checked.

[e-mail](bug opener), if you use Tb with "Show only subscribed folder" unchecked, is checking "Show only subscribed folder" a workaround of your case?

Comment 16

[u442193]

(In reply to WADA from comment #15)
> > Bug summary : 
> >   saved search / virtual folders don't remember folders to search for
> >   3rd+ level of folders
> 
> Which option do you use at Server Settings/Advanced?
>   "Show only subscribed folder" is Checked, or Uncheced
> 
> If Unchecked, phenomenon may be one like next.
> Assume Inbox, Inbox/A, Inbox/A/B, Inbox/A/B/C, Inbox/A/B/C/D, ...
> (a) Account is collapsed upon restart
> (b) Tb doesn't request LSUB when "Show only subscribed folder" is Unchecked.
>     Instead, try to use only LIST for all folders.
>     Request upon restart is for only two level below - LIST %/%, LIST
> Inbox/%/%.
>     Upon expansion of Mbox, LIST Mbox/%/% is reuested.
>     And, when collased Mbox is double-clicked, LIST Mbox/% is also requested.
> (c) If server has problem like bug 710507(nothing is returned to LIST
> Mbox/%/%,
>     returns subfolder names only for LIST Mbox/%), lower level folders can
> not
>     be know once folder structure is lost, until a folder is double-click'ed.
> (d) By (b), when Inbox is collapsed upon restart,
>     if folder structure is somehow lost upon restart,
>     Inbox, Inbox/A, Inbox/A/B is known, but Inbox/A/B/C and below is unknown.
>     Problem like (c) easily make it worse.
> (e) "Re-indexing of .msf upon resatrt" in your case may be a phenomenon of
>     "folder structure is somehow lost upon restart" what I called in (d).
> (f) When a folder poined in virtualFolders.dat is not found, Tb removes the
>     missing folder from search target folder list in virtualFolders.dat.
> (g) If "Show only subscribed folder" is Checked, LSUB is used upon restart
> of Tb.
>     Because only existent and subscribed folder can be selected upon
>     search target folder request for a Virtaul Folder(Search Folder),
>     folder used by a Virtual Folder is usually contained in LSUB response.
>     So, no problem occurs when "Show only subscribed folder" is Checked.
> 
> [e-mail](bug opener), if you use Tb with "Show only
> subscribed folder" unchecked, is checking "Show only subscribed folder" a
> workaround of your case?

I assume the above settings are for IMAP accounts? Affected folders are all located in local folders on my local hard drive.

Comment 17

[u442193]

Issue is still pending. Now Windows 7x64, TB ESR 17.0.10

Comment 18

[u442193]

I have made some new tests today. Re-indexing takes place when more than 500 folders are in the virtual folder. For all folders over 500 in the virtual folder MSF-files are erased and newly generated every time I open the virtual folder.

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to bugzilla from comment #16)
> Affected folders are all located in local folders on my local hard drive.
(In reply to bugzilla from comment #18)
> I have made some new tests today. 
> Re-indexing takes place when more than 500　ｆolders are in the virtual folder.
> For all folders over 500 in the virtual　folder MSF-files are erased and newly generated
> every time I open the　virtual folder.
If so, it means ;
  Problem like bug 905576 occurs and rebuild-index happens,
   (a) on *ALL* 500 folders which are defined as search target folder of a virtual folder,
   (b) *EVERY TIME* you open the virtual folder,
   (c) even when file for mail folder is located on local HDD instead of SMB1 file server,
  in your environment.

On Win 7, standard mail directory location n Tb is as follows.
  C:\Users\<Win UserID>\AppData\Roaming\Thunderbird\Profiles\<ProfName>\Mail\<ServerName>
(Q1) Is your Tb's mail directory/file defined as MS Win's "Shared Folder" and "Offline File"?
> http://windows.microsoft.com/en-us/windows7/understanding-offline-files
>   Understanding offline files
> http://technet.microsoft.com/en-us/library/cc755136.aspx
>   Configure Offline Availability for a Shared Folder

> For all folders over 500 in the virtual folder MSF-files are erased and newly generated every time I open the virtual folder.
(Q2) Actually "MSF-files are *ERASED* and *NEWLY GENERATED*" and actually "*EVERY TIME* you open the virtual folder" and actually "*ALL* folders over 500 in the virtual folder"?
Even if rebuild-index(same as "Repair Folder") occurred on many folders upon "virtual folder open" due to lost or broken .msf file, it should be only once for a mail folder.
And, if outdated-msf condition, as I wrote in bug 905576, rebuild-index doesn't occur upon "virtual folder open", and rebuild-index should happen upon "explicit folder open"(switch to the folder by click folder at folder pane) or upon "open Folder Properties via context menu of mail folder".

Comment 20

[WADA:World Anti-bad-Duping Agency]

Sorry, I misunderstood "all folders over 500". It was "all of 513-th folder to last(more than 500) folder which has highest folder name".
(Q3) Is following correct?
     VF-1 : search target = 1-st to (NN)-th folder, where NN>500
            => problem surely occurs
     VF-2 : search target = 1-st to (NN-1)-th folder, last folder is removed from VF-1
            => problem never occurs

Comment 21

[WADA:World Anti-bad-Duping Agency]

Attachment: virtual-folder-scope-sort.html

HTML/JavaScript to sort "scope=" line in virtualFolders.dat for ease of testing.

(In reply to bugzilla from comment #14)
> I have a little more than 512! folders. Fixing will not help, because as
> described above, if I put the same folder that is now the last in the TB
> sort order, together with all its subfolders alphabetically as first folder,
> TB will not try to reindex this folder any more. It will then try to reindex
> the "newly" last folder of the sort order.
> 
> It will always be that last folders in the alphabetical sort order of TB.

Because "string order by JavaScript's Array.sort()" != "Tb's order of folder", attaced HTML/JavaScript won't be useful for testing. However, "Removing dups in scope= line" and "Sorted order of URI in scope= line" may help testing.

As written in bug 905576, if "rebuild-index" happens, NSPR log by MSGDB:5 is written for %temp%\MozillaMailnews\<MailFolderName>.msf (if Win-XP, 
C:\DOCUME~1\<Win User Name>\LOCALS~1\Temp\MozillaMailnews\<MailFolderName>.msf).
In your test, "getting NSPR log with MSGDB:5" may be helpful.

Comment 22

[u442193]

(In reply to WADA from comment #19)
> (In reply to bugzilla from comment #16)
> > Affected folders are all located in local folders on my local hard drive.
> (In reply to bugzilla from comment #18)
> > I have made some new tests today. 
> > Re-indexing takes place when more than 500　ｆolders are in the virtual folder.
> > For all folders over 500 in the virtual　folder MSF-files are erased and newly generated
> > every time I open the　virtual folder.
> If so, it means ;
>   Problem like bug 905576 occurs and rebuild-index happens,
>    (a) on *ALL* 500 folders which are defined as search target folder of a
> virtual folder,
>    (b) *EVERY TIME* you open the virtual folder,
>    (c) even when file for mail folder is located on local HDD instead of
> SMB1 file server,
>   in your environment.
> 
> On Win 7, standard mail directory location n Tb is as follows.
>   C:\Users\<Win
> UserID>\AppData\Roaming\Thunderbird\Profiles\<ProfName>\Mail\<ServerName>
> (Q1) Is your Tb's mail directory/file defined as MS Win's "Shared Folder"
> and "Offline File"?
> > http://windows.microsoft.com/en-us/windows7/understanding-offline-files
> >   Understanding offline files
> > http://technet.microsoft.com/en-us/library/cc755136.aspx
> >   Configure Offline Availability for a Shared Folder
> 
> > For all folders over 500 in the virtual folder MSF-files are erased and newly generated every time I open the virtual folder.
> (Q2) Actually "MSF-files are *ERASED* and *NEWLY GENERATED*" and actually
> "*EVERY TIME* you open the virtual folder" and actually "*ALL* folders over
> 500 in the virtual folder"?
> Even if rebuild-index(same as "Repair Folder") occurred on many folders upon
> "virtual folder open" due to lost or broken .msf file, it should be only
> once for a mail folder.
> And, if outdated-msf condition, as I wrote in bug 905576, rebuild-index
> doesn't occur upon "virtual folder open", and rebuild-index should happen
> upon "explicit folder open"(switch to the folder by click folder at folder
> pane) or upon "open Folder Properties via context menu of mail folder".

Q1 complete profile is located on drive D:, not like bug 905576 ALL directories but the ones above 500
Q2 YES

Comment 23

[u442193]

(In reply to WADA from comment #20)
> Sorry, I misunderstood "all folders over 500". It was "all of 513-th folder
> to last(more than 500) folder which has highest folder name".
> (Q3) Is following correct?
>      VF-1 : search target = 1-st to (NN)-th folder, where NN>500
>             => problem surely occurs
>      VF-2 : search target = 1-st to (NN-1)-th folder, last folder is removed
> from VF-1
>             => problem never occurs

Yes, correct.

Comment 24

[u442193]

Gentlemen,

the issue seems to be resolved. I would like to describe briefly what I think solved the problem.

For some reason I was wondering why smart folders / unified folders did not work on my office pc. It did work on all my home pcs. So I searched for a solution to resolve the smart folder issue (exit thunderbird > delete the respective smart folders > restart thunderbird). Now after smart folders were working again I noticed that all problems connected with virtual folder and index rebuild are also gone. In the meantime I am using TB24.2.

Comment 25

[Wayne Mery (:wsmwk)]

(In reply to bugzilla from comment #24)
> I ... (exit thunderbird > delete the
> respective smart folders > restart thunderbird). Now after smart folders
> were working again I noticed that all problems connected with virtual folder
> and index rebuild are also gone. In the meantime I am using TB24.2.

so in the past ~2 years you had never done this before?

I would consider this to be a workaround, not a solution.  So I suggest we don't close this bug report pending further investigation, and correlation to other bug reports and support requests

Comment 26

[u442193]

Hi there,

I am still using virtual folders and I still have the issue described above. We are now in 2016 and we are talking W10x64 and TB45.2.0.

in local folders I have approx. 600+ folders
I generate a new search in local folders e.g. status = marked
at that time some or all folders are indexed
I get a certain number of entries in the search window here 142
I save as virtual folder
I open the virtual folder
at that time some or all folders are indexed again
I get a different value of search results in the virtual folder than I had before in the search window here 117
I restart TB
I open the virtual folder
at that time some or all folders are indexed again
I get a different value of search results in the virtual folder than I had before in the search window e.g. 118/121/124/135/141
if I manually redo the search in local folders I get the initial result, here 142

Another interesting thing is to generate two virtual folders that do the same search

every time you reopen TB they will give different results
sometimes results between both virtual folders will be the same, sometimes not

Regards,
Thomas

Comment 27

[u442193]

Just in case this helps:

When accessing a virtual folder I get tons of this error message?

Zeitstempel: 08.07.2016 14:28:15
Fehler: DEPRECATION WARNING: Encoding to non-UTF-8 values is obsolete
You may find more details about this deprecation at: http://bugzilla.mozilla.org/show_bug.cgi?id=790855
resource://gre/components/mimeJSComponents.js 438 MimeConverter.prototype.encodeMimePartIIStr_UTF8

Regards,
Thomas

Quelldatei: resource://gre/modules/Deprecated.jsm
Zeile: 79

Comment 28

[:aceman]

That looks interesting. Can you file it as a new bug? It does not seem related to the virtual folder manager itself.

Comment 29

[u442193]

However, I can only reproduce this bug when using virtual folders.

Comment 30

[u442193]

Here comes some more information;

After starting Thunderbird I access a virtual folder and it will take about one minute until the final number of search results appear.

In the meantime it says in the status line at the bottom of the screen:

Searching...
Generating summary file for ...

If you exit the virtual folder prior to finishing the complete search process you get a random number of results. In any case you will get random number of results.

If you then start a manual search with the same search conditions that you have in the virtual folder it will give you the correct result. You will also see the same messages in the status line of the search windows as described above. If you then open the virtual folder it will also give you the correct result.

Comment 31

[:aceman]

Kent, is the "Generating summary file for ..." normal operation? Or does it mean msf was unexpectedly corrupt and needs to be rebuilt?

Comment 32

[Kent James (:rkent)]

I don't know the answer to that. I also could not find that string in the code. Is that a personal English translation of text in another language?

Comment 33

[u442193]

Installed the us version the see the original status messages:

Searching...
Building summary file for ...

Regards,
Thomas

Comment 34

[u442193]

After four years wait I would like to request to close this issue. Obviously nobody else has this issue and it can be solve by just not using more then 500 folder. I moved all folders > 500 to another location on an external drive and now all works well. Ok this is how software development works!

Merry Christmas.

Comment 35

[[e-mail]]

Hello there,

I would like to bring this issue back again. I have now generated a nice test setup for the developers that you can add with local folders. Whoever is responsible, please contact me. I will send the file.

Regards,
Thomas