522222 - New added or deleted tags not refreshed in gloda index till restart, tags don't show in search results or autocomplete

Status: NEW

(Thunderbird :: Search, defect)

Description

[ovidiu]

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5pre) Gecko/20091013 Shredder/3.0pre

A. 
-create a new tag by tag button /new or options ...
-tag some msg with it
-search in "search all msg" mode for the tag

result 
*it does not propose the new tag in the dropdown
Have to restart tb

expected 
immediate (or almost) including the tag in the search logic

B. [variation for delete considering the str above]
*create tag, tag msg, eventually restart tb
+
-options, display, delete the respective tag
-serch for it in "search all msg"

*it will still propose this old, deleted tag name in the dropdown
*It even finds the msg that used to be tagged with the old one!
I tried various actions for a while and did this again and again looking for another way to overcome this

Comment 1

[ovidiu]

Attachment: exmpla of added and deleted tag with search over them

Comment 2

[Ludovic Hirlimann [:Usul]]

confirming

Comment 4

[Wayne Mery (:wsmwk)]

Another variation on the theme
1. create tag
2. tag msg
3. search to get msg in search results

tag isn't listed in facets 

datalossy :) so nominating

Comment 5

[Mark Banner (:standard8)]

Whilst it obviously isn't ideal, I don't think this blocks a 3.0.x release, wanted though if the patch is reasonable for a stable branch.

Nominating for 3.1, again, I'm not quite sure we'd block on it but we'll put it down for consideration.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Not going to happen in the 3.1 timeframe.

Comment 7

[WADA:World Anti-bad-Duping Agency]

If local mail folder, tag is held in X-Mozilla-Keys: header(and saved in .msf).
If IMAP folder, it's not held in offline-store. It's held as flag(keywaord) of IMAP at IMAP server and is saved in .msf.
Is header name of X-Mozilla-Keys: or string in the header indexed by Gloda which is for full text search of mail body text(and for some data in key headers such as subject)?
Is data of other header such as Received: header indexed by Gloda which is for full text search?
Please note that Gloda's core SQLite3 tables(fts3 virtual table) are created by next SQL statements.
> CREATE VIRTUAL TABLE messagesText USING fts3(tokenize mozporter,
> subject TEXT, body TEXT, attachmentNames TEXT, author TEXT, recipients TEXT)
> CREATE VIRTUAL TABLE conversationsText USING fts3(tokenize mozporter, subject TEXT)
Probably, attachmentNames==file name of attachments, author==From: (Sender:?), recipients==To:, CC:, etc.

(In reply to comment #4)
> datalossy :) so nominating
(In reply to comment #5)
> Nominating for 3.1, again, (snip)

Does it mean that enhanement of "indexing of other header data" is planned or you are requesting it? 
Or enhancement of mixed search(current Gloda search + mailnews/base/search) by "Global Search or Search All Messages" is planed?

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

WADA, you don't need to reverse engineer how gloda works from the database tables; there is documentation in the source.

I would avoid cluttering the bug with analysis unless you are going to provide a fix... if I end up fixing it, I already know how to fix it, and if someone else volunteers I can point them at the spots in the code and answer whatever questions they have.

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #8)
> WADA, you don't need to reverse engineer how gloda works from the database
> tables; there is documentation in the source.

It's not reverse engineering. I saw some Gloda source which defines tables and I saw really issued SQL statements using Gloda's debuging feature, and I'm "[e-mail]" in next problem report to SQLite3 developer.
> http://code.google.com/p/sqlite-manager/issues/detail?id=444
For me, pasting of data displayed by SQLite Manager or sqlite3.exe was easiest way to put data about current implemetation in this bug than pointing source code.
My questions were;
(A) Is request of tag data support to "Indexer of Gloda" proper?
    Is request of tag data support to "Global Search" proper?
(B) If enhancement or improvement around search of tag data will be done,
    should it be what kind of solution, will it be what kind of solution?
As comment #0 was report on Global Seacrh and tag data, before I ask (B) to you, I asked about (A) to Wanye and Mark and you. As I wanted to know about future design which should be done before start of implementation by providing patch, I wanted to ask you about (B).

Above was before I aware of duping of bug 523327 by Ludo...
Problem was not in search itself. Problem was after Global Search only.
I couln't understand meaning of "propose", "dropdwon" in next until I read bug 523327.
> *it does not propose the new tag in the dropdown
> *it will still propose this old, deleted tag name in the dropdown
I misunderstood bug as I thought it's for search problem.
> *It even finds the msg that used to be tagged with the old one!
Sorry for misunderstanding and cluttering.

Quick check with Glodaquilla.

(Case-1) pre-test. what mail hits when duplicated mails exist.
(1) copy a mail in a local folder from other account(F1, glodaid=20)
    copy the mail to F2, glodatd=20, copy the mail to F3, glodaid=20.
(2) Repeat .msf detetion, use profile with Tb3.0.4 and Tb3.1pre, etc.
    to force reindex by Gloda.
    => F1/glodaid=21, F2/glodaid=22, F3/glodaid=23 for duplicated mails.
(3) Gloda search, hits one mail only.
    => glodaid=20 is displayed in "Open as list" 
Apparently mail of first assigned docid is displayed as search result.
I suspected similar issue is involved in bug report, as i misunderstood search problem is involved in some phenomena reported to bug.

(Case-2) To understand meaning of "propose", "dropdwon"
(0) tag1, tag2 : already defined before restart of Tb3.1pre.
(1) delete global-messages-db.sqlite, restart Tb3.1pre
    originals : F1A/glodaid=20/tag1, F2A/glodaid=21/tag2, F3A/glodaid=22 
                20, 21, 22 are different mail, has same string in subject
    copies    : F1B/glodaid=30/tag1, F2B/glodaid=31/tag2, F3B/glodaid=32 
(2) create tag3, tag4, add tag3 to 21, add tag4 to 22
    rename tag1 to tag1X, delete tag2
    As Gloda's activity is displayed in console upon tag add/tag remove,
    and as removed/added tag1/tag2 is properly handled by Global Search,
    there is no problem in tag change handling any more.
(3) Gloda search, mail in F1A(20), F2A(21), F3A(22) hits
    At search result display by Gloda
    => 20: still tag1, 21: still tag2 displayed, no tag3, 22: no tag4
    Tags are displayed as expected at tag column in "Open as list".

It looks that problems around tag observed in above quick check is;
(a) Problem of comment#0 : Gloda doesn't detect tag definition change.
(b) If duplicated mail of different glodaid exist, user confuses by his action
    around tag(add/remove/create/delete/rename tag).
    - Duplicated mail exists before first use of Gloda.
    - Duplicated mail is downloaded.
    - Duplicated mail is copied/moved from other account.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Andrew Sutherland (:asuth), do I need to open bug for confusion by user when user generated duplicated mail and changed tag of them?

Comment 11

[WADA:World Anti-bad-Duping Agency]

Question about workaround:
Restart of Tb after tag definition change is only workaround?

Comment 12

[Wayne Mery (:wsmwk)]

(In reply to WADA from comment #11)
> Question about workaround:
> Restart of Tb after tag definition change is only workaround?

yes. 

(In reply to WADA from comment #10)
> Andrew Sutherland (:asuth), do I need to open bug for confusion by user when
> user generated duplicated mail and changed tag of them?

IMO no, not unless adding another tag surfaces a different bug

Comment 13

[Wayne Mery (:wsmwk)]

xref Bug 532836 - let user know if the search keyword is not indexed when "No messages match your search"