Use same syntax for msgFilterRules.dat and VirtualFolder.msf

UNCONFIRMED
Unassigned

Status

MailNews Core
Search
--
enhancement
UNCONFIRMED
5 years ago
5 years ago

People

(Reporter: Ulf Zibis, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0 (Beta/Release)
Build ID: 20131115110311

Steps to reproduce:

Composed some filter rules for virtual folder on Inbox, saved in VirtualFolder.msf, to examine the result.


Actual results:

Cannot use data from VirtualFolder.msf to add composed filter rules into msgFilterRules.dat because of different syntax, e.g.:
VirtualFolder.msf:   ("reply-to",contains,bugs.launchpad.net\)
msgFilterRules.dat:   (\"reply-to\",contains,bugs.launchpad.net\)


Expected results:

Both files should use same syntax, so filter rules could be easily copied from virtual folder filter to message filters.
(Reporter)

Updated

5 years ago
Severity: normal → enhancement
> Customized header definition
>   mailnews.customHeaders = rECEIVED: rEPLY-tO: dATE
> term= in virtualFolders.dat, condition= in msgFilterrules.dat by Tb 24.1.1 for :
>   Date is after ..., dATE begins with ..., rEPLY-TO contains ...
> (A) virtualFolders.dat :
>      terms=AND (date,is after,02-Dec-2013) AND ("Date",begins with,abc)
>            AND ("reply-to",contains,!!!)
> (B) msgFilterrules.dat :
> condition="AND (date,is after,02-Dec-2013) AND (\"Date\",begins with,abc)
>            AND (\"reply-to\",contains,!!!)"
As you say, it looks that different syntax rule is used by "message filter" and "search".
                    (A) Predefined  (B) Customized(special)     (C) Customized(non-special)
 virtualFolders.dat     date            "Date"                      "reply-to"
                        (lower case)    (starts with upper case)    (lower case        )
                        (no quote  )    (with quote            )    (with quote        )
                                        (no escape of quote    )    (no escape of quote)
 msgFilterrules.dat     date            \"Date\"                    \"reply-to\"
                        (lower case)    (starts with upper case)    (lower case        )
                        (no quote  )    (with quote            )    (with quote        ) 
                                        (with quote escaped    )    (with quote escaped)
Difference is :
  - quote is NOT    escaped by \ in virtualFolders.dat
  - quote is always escaped by \ in msgFilterrules.dat

As seen in "keyword for conditions"(term= and condition=), different keyword is used in "message filter" and "search", and, needless to say, different file name is used.
This means that "different parser" and "different generator" is used in "message filter" and "search".

> Expected results:
> Both files should use same syntax, (snip)
Why it can be *SHOULD* even though "message filter" and "search" are different feature?
If your claim is "better to use same syntax" && Severity=Enhancement(a kind of design change request instead of actual flaw in code. you already set this), I agree wth you, because mailnews.customHeaders is shared by two features and because I can't imagine reason why different syntax is mandatory in almost same function of two features.
And, if most important gain by enhancement == "we will be free from chaos" instead of "you can do Copy&Paste", I absoluely agree with you :-)

"difference between date and Date" was/is for internal rule of "lower case only == pre-defined property saved in .msf==MsgDB (numeric value if date)" and "Starts with upper case == message header name (value is string in the message header)", if same name as pre-defined name is defined as official message header name(case insensitive) in mail related RFCs.
I guess ;
  Virtual folder :
    quote is removed by parsing, because it's merely a quoted string.
    So, internal rule of "Starts with upper case or not" is still mandatory and effective.
  Message filter :
    quote is not removed by parsing, because quote is escaped by \.
    In order to avoid chaos in the past by "Starts with upper case or not" rule,
    (many developers were confused due to from vs. From, sender vs. Sender,)
    (status vs. Status, date vs. Date, ..., then wrong code was created.   )
    message filter code started trial of introducing rule of "quoted or not"
    in determination about "message header name or pre-defined property name".
    Even if so, IIRC, "Starts with upper case or not" rule is still effective
    in message filter too.
Component: Filters → Search
OS: Linux → All
Product: Thunderbird → MailNews Core
Hardware: x86_64 → All
I think "same change as one in message filter" is needed in search(virtual folder), so I set Component:Search. If wrong, correct it, please.

By the way, if syntax of "term=" in virtualFolders.dat will be altered, needless to say, following is needed for upward compatibility.
- Conversion of all "term=" lines in existent virtualFolders.dat file.
- Conversion of all existent VirtualFolder.msf file content.
And, once conversion is done, downward compatibility of virtualFolders.dat and VirtualFolder.msf is surely lost as usual.

What do you think about it?
Do you believe that cost of implementing such conversion code is pretty low? ("cost" in this context == required workload, required time etc. of many volunteers)
Is expected gain far greater than estimated cost for implementing?
(requesting to other peoples is pretty easy, however, implementing it is usually hard job)
(Reporter)

Comment 3

5 years ago
(In reply to WADA from comment #1)
> > Customized header definition
> >   mailnews.customHeaders = rECEIVED: rEPLY-tO: dATE
> > term= in virtualFolders.dat, condition= in msgFilterrules.dat by Tb 24.1.1 for :
> >   Date is after ..., dATE begins with ..., rEPLY-TO contains ...
> ....
Thanks for your detailed explanation WADA.
I'm wondering, why you refer to virtualFolders.dat,
because I only see VirtualFolder.msf in my profile.

> > Both files should use same syntax, (snip)
> Why it can be *SHOULD* even though "message filter" and "search" are
> different feature?
> If your claim is "better to use same syntax" ...
I agree with you :-)

> "difference between date and Date" was/is for internal rule ...
Thanks again for your explanation :-)
Do I understand correct? :
Lower case "sender" is already in use as pre-defined property name.

Aside: Do you know of about any more smart possibility (e.g. GUI function or Addon) to copy examined virtual filter rules to message filters, instead of manually copying and converting text from VirtualFolder.msf to msgFilterrules.dat and maybe vice versa?

(In reply to WADA from comment #2)
> - Conversion of all "term=" lines in existent virtualFolders.dat file.
> - Conversion of all existent VirtualFolder.msf file content.
> What do you think about it?
+1 from my side :-)

> Is expected gain far greater than estimated cost for implementing?
Additional gain: Simplified work for copying examined virtual filter rules to message filters.

Aside: From Filter GUI I only can choose an account but not any sub folder as source for the filters.
Would it work correctly, if I copy and then customize a msgFilterrules.dat file to a folder?

BTW: In my Filter GUI "Apply selected filter(s) on:" provides an empty list from the combo box. Is that a bug?
(Reporter)

Comment 4

5 years ago
Can you change from UNCONFIRMED to NEW?
(In reply to Ulf Zibis from comment #3)
> I'm wondering, why you refer to virtualFolders.dat,
> because I only see VirtualFolder.msf in my profile.
What directory in file system of OS do you call by "my profile"?

VirtualFolder.msf(.msf file for Search Folder named VirtualFolder) is "per folder" file, so it's placed under mail directory assciated to an account(server), or under Parent.sbd directory for parent mail folder named Parent.
virtualFolders.dat is "per Thunderbird" file, so it's placed under Profile directory, as prefs.js is placed under Profile directory.
msgFilterRules.dat is "per account(server)" file, so it's placed under mail directory assciated to an account(server).
(In reply to Ulf Zibis from comment #3)
> Do I understand correct? :
> Lower case "sender" is already in use as pre-defined property name.
Lower case "sender" is perhaps not used as pre-defined property name in filter/search.
  IIRC, internal name for "From at filter GUI" was Lower case "sender".
  IIRC, The lower case "sender" was for "mail sender data from From: Sender:".
  IIRC, name for it was changed to Lower case "from", by sorting out internal
  naming convention, during resolving problems due to
  sender(data from From:, Sender:) vs. Sender(message header named Sender:) war.
  I don't know whether current "From at filter GUI"(corresponds to Lower case from)
  is "data from From: and Sender:" or "data from From: only".
  But I may be confused with "column name change from Sender to From" at thread pane.
(Reporter)

Comment 7

5 years ago
(In reply to WADA from comment #5)
> (In reply to Ulf Zibis from comment #3)
> > I'm wondering, why you refer to virtualFolders.dat,
> > because I only see VirtualFolder.msf in my profile.
> What directory in file system of OS do you call by "my profile"?
I was only looking in the accounts tree :-(

> virtualFolders.dat is "per Thunderbird" file, so it's placed under Profile
> directory, as prefs.js is placed under Profile directory.
> msgFilterRules.dat is "per account(server)" file, so it's placed under mail
> directory assciated to an account(server).
Thanks, I now found it. But this means error-prone redundancy of the virtual filter rules?

But why this inconsistency?
So maybe additionally think about:
> - Conversion of all xxx.dat files to place them in same location, "per Thunderbird" or "per account(server)".
(In reply to Ulf Zibis from comment #7)
> Thanks, I now found it. But this means error-prone redundancy of the virtual filter rules?
"Redundancy" is perhaps an "Unforgiven" for Germany, but it's a "What should be loved" for Japanese like me :-)

Definition of "Virtul Folser" is generated/saved in virtualFolders.dat file first.
After it, the "defined Virtual Folder" is changed to "an actual instance of a Virtual Folder" which has associated file named xxx.msf, and "definition of Search Target Folder && Search Condition" is saved in file name.msf for ease of program logic coding.
So, the "an actual instance of a Virtual Folder" is called "(Saved) Search Folder" by current Thunderbird.
I believe current "Search Folder", and current "Unified Folder" which is based on current "Search Folder", is never imaginary "Virtula Folder".

I usually never consider this kind of "duplication" as useless/meaningless "Redundancy".
Do you know ”Adverse effect” by "over normalization" in Relational Data Base world?
"Perfect normalization" surely removes "Redunduncy" perfectly, but it can easily produces "difficulty in program coding by ordinal humen beings like me, due to difficulty in understaning Data Base design which is caused by perfect normlization", and "performane of program code or efficiency of program code/coding" is degraded by the "Perfect normalization", because many code steps is required by the "Perfect normalization" which  Germany like you perhaps always deeply loves :-)
(Reporter)

Comment 9

5 years ago
Nice observation of German and Japanese habits :-)
Hopefully that accurateness prevents us from things like Fokushima, but I'm not really sure.

But I do not really understand, why you need the global virtualFolders.dat file at all, as you don't need it global for the message filters.
BTW: It takes up to 20 sec. to open VirtualFolder->properties here, is that *performance* ?
     In contrast, message filter properties open in <1 sec.

Nevertheless I'm missing your answer about *arbitrary folder-wise" message filters.
I would like to pre-filter messages server-side, to have a better overview when I'm travelling and then I want fine-granulated folder-wise filters via TB, e.g. for moving messages to local folders or for filtering the Sent folder.

And to come back to my original purpose, I think, it should be possible to use non-escaped quotes for msgFilterrules.dat, just use the parser from virtualFolders.dat plus:
- conversion of "terms=" to "condition="
- conversion of "actionValue=", "uri=" to "target="
- add "scope=" to msgFilterrules.dat
(In reply to Ulf Zibis from comment #9)
> But I do not really understand, why you need the global virtualFolders.dat
> file at all, as you don't need it global for the message filters.

Why definition of "Virtual Folder" should be saved in "per folder file named xxx.msf" even though the definition is already held in global/centralized file named virtualFolders.dat?

As for viewing/searching mail data which is locally held, simplest and nearly best way is;
- Hold mail data in global data base, for example, SQL data base, after "normalization",
  for example, covert text data to Unicode, decode base64/quoted-printable, compress
  big attachment data if required, ..., with clever indexing for efficient search.
  Note: Gloda, Global Indexer and Search, is an approach of "clever indexing" which
        uses a part of Google's search technique in Web world.
- When view a mail, "SELECT where(message_key=nnn)", show mail data.
- When search, "SELECT where(xxx=aaa and (yyy=<bbb or zzz contains ccc) ...",
  show list of mails, and if needed, cache search result.
Developrs already tried it(SQLite was used) in Thunderbird, but for performance reason, current MorkDB(.msf) is still used.

Gmail obviously utilizes "centralized mail data base". As seen in design of "mail folder shown via IMAP == Gmail Label", there is no actual traditional "mail folder" in Gmail. Gmail simply provides several ways to access mail data in "centralized mail data base", for example, via Web interface, via protocol named IMAP, via protocol named POP3, via Gmail's API(http access) and so on.
See bug 721316 and its dependency tree for "unique key assigned to a mail in Gmail" called X-GM-MSGID(and X-GM-THRID), please.

> I'm missing your answer about *arbitrary folder-wise" message filters.

"Thunderbird wise filter", "per folder filter", "multiple and selectable filter" etc. are already requested by many users. Search bugzilla.mozilla.org by yourself with your hand, please.

> In contrast, message filter properties open in <1 sec.
It's only when "msgFilterRules.dat is already read because filter is used by message filter upon download or message filter is opened once", isn't it?
Try following.
1. Create new profile, define dummy news account in order to define "Local Folders".
   Create pretty big msgFilterRules.dat file using small Script or Text Editor.
   Because pseudo "unique key" is "name=" only, it's pretty easy.
   IIUC, "name=" is merely a label, so I think all rules can have same "name=". 
2. Restart Tb
3. Open message filter
4. Close message filter, and open message filter again.
Bug 571419 is already fixed in Tb 17, but time to read msgFilterRules.dat is not ZERO, and it takes relatively long if msgFilterRules.dat is big. So step 3 takes a while in ordinal environment, even when disk cache is fully/efficiently utilized by OS.
Once msgFilterRules.dat is read, data is held in memory, so "physical read of file" is skipped. Then step 4 is quicker than step 3.

> BTW: It takes up to 20 sec. to open VirtualFolder->properties here, is that *performance* ?

IIUC, "Open folder properties of Search Folder named XXX" is never "simply read XXX.msf file and simply show filtering rule part in XXX.msf file content".
Is there difference among following?
(1) After restart of Tb, when other than Search folder is selected at folder pane,
    Start Process Monitor and watch Tb's file access,
    Explicitly open the Search folder(left click),
    Keep backup of log file.
(2) After restart of Tb, when other than Search folder is selected at folder pane,
    Start Process Monitor and watch Tb's file access,
    Show folder properties of the Search folder(open context menu by right click),
    Keep backup of log file.
(3) Force "outdated msf condition" on a search target folder,
    and show Folder Properties of Search Folder.
(3-1) A search target folder of Search Folder named XXX = FolderX.
      Edit file named FolderX(not FolderX.msf) by Text Editor,
      change file size by adding string or row, or removing string or row, and save.
(3-2) Enable Tb's logging with MSGDB:5, and restart Tb.
      After restart of Tb, when other than Search folder is selected at folder pane,
      Start Process Monitor and watch Tb's file access,
      Show folder properties of the Search folder(open context menu by right click),
      Keep backup of log files.

Is Tb's access to C:\ ... \Temp\MozillaMailnews\FolderX.msf logged by MSGDB:5 by (3-2)?
> nsMsgDatabase::Open(C:\ ... \LOCALS~1\Temp\MozillaMailnews\FolderX.msf, ...
(See bug 905576 for this log when rebuild-index is invoked by Tb.)
Is there difference in Tb's file access among above three cases?
> I think, it should be possible to use non-escaped quotes for msgFilterrules.dat,
> just use the parser from virtualFolders.dat (snip)
I also think so, because it's far easier to read and understand for us.
But purpose of "escaping quote in msgFilterRules.dat" is "to keep quote after parsing condition= line", and this "keeping quote after parsing" is done in order to avoid problems and chaos due to "Lower case name==pre-defined and Upper case Name==header" rule.
I believe rule of "non-quoted==pre-defined and quoted==header" is far safer. Because message header is case insensitive, decision on someting around message header based on "case of ascii string" is pretty dangerous.
(Reporter)

Comment 12

5 years ago
(In reply to WADA from comment #10)
> IIUC, "Open folder properties of Search Folder named XXX" is never "simply
> read XXX.msf file and simply show filtering rule part in XXX.msf file
> content".
Why?
Even if there are 50 filter rules, it's only 1..5 KByte to parse and could be read in one chunk from the file system, regardless if retrieved from virtualFolders.dat or xxx.msf. IMHO both could be done in a fraction of a second.

>     Explicitly open the Search folder(left click),
Obviously this cold take some time, as all concerning mail headers must be read and examined.

>     Show folder properties of the Search folder(open context menu by right
> click),
But this should be fast, as justified above.

(In reply to WADA from comment #11)
> I also think so, because it's far easier to read and understand for us.
:-)

> But purpose of "escaping quote in msgFilterRules.dat" is "to keep quote
> after parsing condition= line",
This is needed because the argument of condition= is just quoted, but IMO this is not neccessary as you can see for terms= in virtualFolders.dat and/or xxx.msf

> I believe rule of "non-quoted==pre-defined and quoted==header" is far safer.
I agree, but this does not justify to generally quote all arguments in msgFilterRules.dat.
Following is surely undestandable for all programmers.
- You can't simply do copy string in msgFilterRules.dat to virtualFolders.dat,
  or vice versa, using Text Ediror.
- Current escaped quote in condition= line is sufficiently ugly
  and it makes "our reading msgFilterRules.dat" slightly hard.
I think there is no need to repeatedly request "Tb's code change which you want" in *BUG* at B.M.O.

Note:
I also think if syntax were "Header(X-Mozilla),contains,'text "XX" string' AND Category(Status),is,Read AND..." like one...
Further, I think if rules were saved in Relational Data Base or Non-SQL Data Base or Object Store...
I think if search were done by Data Base access method such as "Select Where(a=A and (b!=c or d<e) ) Order by subject"...
"rule"/"search" in above is applicable to Address Book, account definition, thread pane column setting, folder properties etc.

> > I believe rule of "non-quoted==pre-defined and quoted==header" is far safer.
> I agree, but this does not justify to generally quote all arguments in msgFilterRules.dat.
Even if so, problem is merely "you can't simply do copy string in msgFilterRules.dat to virtualFolders.dat or vice versa using Text Ediror" only, isn't it?
Why "you can't simply do copy string in msgFilterRules.dat to virtualFolders.dat or vice versa using Text Ediror" can pretty easily justify required workload of developers who are volunteers?
(In reply to Ulf Zibis from comment #12)
> (In reply to WADA from comment #10)
> > IIUC, "Open folder properties of Search Folder named XXX" is never "simply
> > read XXX.msf file and simply show filtering rule part in XXX.msf file  content".
> Why?
Because;
  "Folder properties of Search Folder" !== 
  "filtering rule text + search target folder uri string" which is saved in file named .msf

Before presenting currently defined "filtering rule text and search target folder uri" as "propery of the search folder", consistency check, for example, existency check of search target folder, virtualFolders.dat content check which is centralized Virtual Folder definition, are needed.
To achive this, nealy same process as "Saved Search folder open" is perhaps needed, although "re-indexing of mails in the search target folder from index data saved in all SearchTargetFolder.msf files" is not executed if "Show folder properties" is requested via Right click without selecting the Search Folder at folder pane.
IIUC, this is different from ordinal mail folder.
If ordinal mail folder, folder properties is a part of data saved in file named xxx.msf.
This can perhaps be done without full "Message data base open of the folder" process in which "read all mail meta data of all mails saved in xxx.msf and re-construct Messaage data base in memory" is involved.

You can perhpas see "rebuild-index of a broken search target folder" by shownig folder properties of saved search folder, without selecting search folder at folder pane(left click, explicite folder open, is not requested), without folder selection/explicit folder open of the broken search target folder at folder pane.
"Corrupting a SearchTargetFolder.msf" is pretty easy.
  Open SearchTargetFolder.msf by Text Editor, Delete all data, Put random text, Save.

> >     Explicitly open the Search folder(left click),
> Obviously this cold take some time,
> as all concerning mail headers must be read and examined.

Accurately, "What is read by it" is not mail headers.
It's MsgDBHdr data(meta data of a mail), which is extracted from message headers and saved in A_SearchTargetFolder.msf, of all messages in A_SearchTargetFolder in all A_SearchTargetFolder.msf files. After mail meta data is read from all A_SearchTargetFolder.msf files, mail meta data is written to MsgDB for SearchFolder which will be saved in SearchFolder.msf. This is currently similar to "simple copy of MsgDBHdr with assignin new MessageKey", so there is problem in "Threaded display of mails" in Virtual Folder with multiple search target folder(Search Folder, Unified Folder etc.)
(Reporter)

Comment 15

5 years ago
(In reply to WADA from comment #13)
> Following is surely undestandable for all programmers.
> - You can't simply do copy string in msgFilterRules.dat to
> virtualFolders.dat,
>   or vice versa, using Text Ediror.
> - Current escaped quote in condition= line is sufficiently ugly
>   and it makes "our reading msgFilterRules.dat" slightly hard.
> I think there is no need to repeatedly request "Tb's code change which you
> want" in *BUG* at B.M.O.
I respect your opinion. Therefore this bug is marked as enhancement.

> > > I believe rule of "non-quoted==pre-defined and quoted==header" is far safer.
> > I agree, but this does not justify to generally quote all arguments in msgFilterRules.dat.
> Even if so, problem is merely "you can't simply do copy string in
> msgFilterRules.dat to virtualFolders.dat or vice versa using Text Ediror"
> only, isn't it?
Well, I would alternatively like a GUI function for exporting the filter rules of a virtual folder, after its examination, to a real message filter. Unless this is not provided (could be another enhancement bug), it would be convenient for experienced users to simply could copy the condition=xyz string.

> Why "you can't simply do copy string in msgFilterRules.dat to
> virtualFolders.dat or vice versa using Text Ediror" can pretty easily
> justify required workload of developers who are volunteers?
On one hand this is correct.
On the other hand, maintaining and hosting 2 different code snippets for basically the same job, i.e. persisting same structured filter rules, is another (work)load.
I'm afraid, even the code for the 2 filter editor GUIs is, at least partially, duplicated in TB's code base.
(Reporter)

Comment 16

5 years ago
(In reply to WADA from comment #14)
> Before presenting currently defined "filtering rule text and search target
> folder uri" as "propery of the search folder", consistency check, for
> example, existency check of search target folder, virtualFolders.dat content
> check which is centralized Virtual Folder definition, are needed.
In my case, there is only 1 search target folder, "Inbox", where existency check should be fast.
It currently contains ~70 filter definitions, Inbox.msf contains ~20.000 mail headers.
When editing these filter definitions, I experienced an increase of GUI opening time the more filter definitions were added.
Consistency check between virtualFolders.dat and SearchFolder.msf is the culprit upon redundancy ;-)

> IIUC, this is different from ordinal mail folder.
> If ordinal mail folder, folder properties is a part of data saved in file
> named xxx.msf.
> This can perhaps be done without full "Message data base open of the folder"
> process in which "read all mail meta data of all mails saved in xxx.msf and
> re-construct Messaage data base in memory" is involved.
Hm, but why should this be done for just opening the "Search Folder Properties" GUI?

> > >     Explicitly open the Search folder(left click),
> > Obviously this cold take some time,
> > as all concerning mail headers must be read and examined.
> Accurately, "What is read by it" is not mail headers.
This is obviously clear, I just formulated my statement "briefly".
(In reply to Ulf Zibis from comment #16)
> Hm, but why should this be done for just opening the "Search Folder Properties" GUI?
I don't know. I'm merely talking about "what current Tb does do perhaps actually".
If I were mail application developer, I use SQL Data Base, DB2 if DB2 is free and can run on my PC, because mail data can be acccesed by simple "Select Where Order By" without using complex MorkDB, without using Unix Mbox format file which I dislike, and without using C++ language in which I can't write code :-)

> In my case, there is only 1 search target folder, "Inbox", where existency
> check should be fast.
You look to forget following;
  Entire data in SearchFolder.msf, which was saved by previous open/close of the
  SearchFolder, is perhaps read and Message Data Base for the SearchFolder is perhaps
  re-constructed in memory first, before start of any activity on the Search Folder
  including process for "display of Folder Property of the Search Folder".

> It currently contains ~70 filter definitions, Inbox.msf contains ~20.000 mail headers.
> When editing these filter definitions, I experienced an increase of GUI
> opening time the more filter definitions were added.

What is definition of the "GUI opening time"?
 From click "Property" at context menu of the search folder
   to "Folder Properties" panel open(==search rule definition panel open)?
Is delay between "search rule defiition panel open" to "completion of all filter rules display in panel" involved?

O( number of mails in search target folder )? (single search target folder)
O( number of rules in a search folder )? O( "number of rules" **2 )?

Get Tb log and Process Monitor log for small SearchFolder and small SearchTargetFolder, and check log.
  Tb log : timestamp,MSGDB:5,DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5
           MSGDB:5,DOMLeak:5,DocumentLeak:5,nsDocShellLeak:5 (if DebugView is used)
     MSGDB:5    track Message Database open/close
     ...Leak:5  track open/close of panel, alert window etc.
  Process Monitor : filter
     Path contains \SearchFolder
     Path contains \SearchTargetFolder
     Path contains virtualFolders
     Path contains \MozillaMailnews\ (accessed if rebuild-index happens)
     Path contains ...\Temp\ (%temp% directory)
To reduce workload of log contet check, get log with new Tb profile, with dummy news account only(to force creation of "Local Folders" account).

> Consistency check between virtualFolders.dat and SearchFolder.msf is the
> culprit upon redundancy ;-)
Needless copy of filtering rule in SearchFolder.msf is redundancy :-)
(Reporter)

Comment 18

5 years ago
(In reply to WADA from comment #17)
> What is definition of the "GUI opening time"?
>  From click "Property" at context menu of the search folder
>    to "Folder Properties" panel open(==search rule definition panel open)?
This takes about ~6 sec.

> Is delay between "search rule defiition panel open" to "completion of all
> filter rules display in panel" involved?
Yes, in sum I have to wait ~20 sec. before I can start entering a new filter rule.

Am I correct, that by "Saved search folder" you mean the same that I originally specified by "Virtual folder"?
(In reply to Ulf Zibis from comment #18)
> Am I correct, that by "Saved search folder" you mean the same
> that I originally specified by "Virtual folder"?

You are correct.
Concept of "Virtual Folder" was implemented as current "Search Folder".
However, implementation of UI to define "filtering rule"(== Current "Folder Properties" of Search Folder, Unified Folder or Smart Folder) was postponed.
Initial way to define "filtering rule of Search Folder" was only "Do search, then save the search rules as Search Folder"(this is still supported), so it was called "Saved Search Folder".
Because Tb already has explicit way to define "Search Folder"(menu of File/New/Saved Search), it's now caled "Search Folder" simply, although "Saved Search Folder" or "Saved Search" is still used as seen in menu of "File/New/Saved Search".
And, as you can see via "Folder Properies of Unified Folder in Unified Folder View", "Unified Folder" is essentially merely a "Search Folder" for which Tb automatically sets "Search Target Folders" and condition of "All mails in the Search Target Folders".
(In reply to Ulf Zibis from comment #18)
> (In reply to WADA from comment #17)
> > What is definition of the "GUI opening time"?
> >  From click "Property" at context menu of the search folder
> >    to "Folder Properties" panel open(==search rule definition panel open)?
> This takes about ~6 sec.
> > Is delay between "search rule defiition panel open" to "completion of all
> > filter rules display in panel" involved?
> Yes, in sum I have to wait ~20 sec. before I can start entering a new filter rule.

If so, major problem is performance problem in "showing filtering rules after getting data of filtering rules for the Search Folder from SearchFolder.msf and/or virtualFolders.dat", isn't it?
Is time between "search rule defiition panel open" and "completion of all filter rules display in panel" O( number of rules )? Or O( "number of rules"**2 )?
Does the time depend on height of "search rule defiition panel"(number of conditions shown in the panel at same time)?
(In reply to Ulf Zibis from comment #15)
> > Why "you can't simply do copy string in msgFilterRules.dat to
> > virtualFolders.dat or vice versa using Text Ediror" can pretty easily
> > justify required workload of developers who are volunteers?
> On one hand this is correct.
> On the other hand, maintaining and hosting 2 different code snippets for
> basically the same job, i.e. persisting same structured filter rules, is
> another (work)load.
> I'm afraid, even the code for the 2 filter editor GUIs is, at least
> partially, duplicated in TB's code base.

I agree with you, if major reason is never "you can't do copy&paste".

It's different but pretty similar to following problem.
  "Save As of attatchment via Context Menu" and "Save As of attachment via double click"
  uses differnt code path, so issue like "automatically selected save target directory is
  different" pretty easily occurs.
  Why "Save As of attatchment via Context Menu" and "Save As of attachment via double
  click" should be different?
  Reason why such mismatch occurs is simply;
    "Save As of attatchment via Context Menu" and "Save As of attachment via double click"
    was developed by different people at different time :-)
Consolidation of code is already requested.

Why almost same function in search and filer should have different spec?
(Reporter)

Comment 22

5 years ago
(In reply to WADA from comment #20)
> If so, major problem is performance problem in "showing filtering rules
> after getting data of filtering rules for the Search Folder from
> SearchFolder.msf and/or virtualFolders.dat", isn't it?
To be precise: after the 6..8 sec. to open the panel, the content is displayed immediately, but it takes another ~30 sec. to become responsive, e.g. scrolling has an effect, and CPU-score falls down from 100 to ~0 %. Even after that time, scrolling is still bumpy.

> Is time between "search rule defiition panel open" and "completion of all
> filter rules display in panel" O( number of rules )? Or O( "number of
> rules"**2 )?
I would say it's O( number of rules ).

> Does the time depend on height of "search rule defiition panel"(number of
> conditions shown in the panel at same time)?
Don't think so, but it's difficult to test, as if I close it from big enlargement, it again opens as small as ever.
You need to log in before you can comment on or make changes to this bug.