1348662 - repair folder does not initiate new Gloda indexing

Status: UNCONFIRMED

(Thunderbird :: Search, defect)

Description

[klaus]

User Agent: Mozilla/5.0 (Windows NT 10.0; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20170302120751

Steps to reproduce:

I would expect a folder repair to initiate a new Gloda indexing of that folder. It does not.


Actual results:

load an addon that blocks Gloda indexing (not intentionally ...).
Change a few tags. Search for tags in a gloda query, the new ones are not found.

Deactivate addon, repair the folder. Gloda still does not find the new tags (= not reindexed?).

Delete the gloda sqlite db. restart TB, now Gloda indexing is ok. Conclusion: repair does not schedule reindexing.

If this is as expected, I would make this a feature request: folder repair should schedule reindexing. If I suspect the folder list to be bad, Gloda could be bad as well.

Comment 1

[Wayne Mery (:wsmwk)]

> load an addon that blocks Gloda indexing (not intentionally ...).
names of addon(s) please?

> I would expect a folder repair to initiate a new Gloda indexing of that folder. It does not.

I'm not an expert on "Repair Folder", so I'm unsure what the defined behavior is for all circumstances. But it seems to me that expected behavior is a redownload of the folder is not needed in all cases. And I believe a redownload is required to retrigger a reindexing of gloda. Note, Repair Folder's history (and the wording in properties) has nothing to do with gloda, and everything to do with the contents of the folder as it exists on the local machine (cache). 

I tested repair on my Sent folder. The gloda dirty bit was flipped on all the messages, and the gloda ID did not change on any messages - probably because none of the messages were redownloaded, hence no message uuid changes.

Aceman may have some insight.

Comment 2

[klaus]

more details: use glodaquilla and this query to return the number of tagged messages and a collection:

 let tagArray = MailServices.tags.getAllTags({});
 query.tags(...tagArray);

Whenever tags are set/removed, I can verify in glodaquilla, and the collection changes accordingly on a new query.

Load replymanager addon. Change a few tags. Mesagewindow reflects the new tags. Gloda query does not, the collection stays as it was before loading the addon.

Remove addon. Gloda query is still unchanged and does not reflect changes in tags. My idea was to repair the folder, hoping that would start a reindex. It does not, Gloda query is unchanged and still wrong.

Delete sqlite db for gloda. Now, messges are reindexed and gloda query is ok.

Problem: upon finding inconsistencies, a user might have the idea to repair, but not to delte the sqlite db. So it would be good if repair would initiate a reindeing for Gloda.

On my TB (45.8.0), repair does not flip the dirty and activity manager does not indicate reindexing (if db is deleted, I see the indexing reported there). Only addons installed are lightning and gldaquilla.

Comment 3

[Wayne Mery (:wsmwk)]

wada's comments in bug 1099275 are actually a great intro to the workings of the offline store and Repair Folder.

Comment 4

[:aceman]

Repair folder is mainly intended to write a new mbox file out of the message existing in the folder and also create a new .msf file (database of the message properties) by reading the raw messages again. This is to be used when the old .msf file somehow gets broken and does not properly represent the messages in the mbox file.
I don't know if it triggers gloda rebuild or any other action there. Maybe asuth could know.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

I don't remember offhand what events rebuilding generates.  I think the correct sequence events would be "delete folder (non-recursive? that's tricky if it has sub-folders), add folder, generate add events for each message as they're re-parsed".  I think it's probably a rebuild folder bug if it doesn't generate events, but I wouldn't rate it a super-high priority.

From klaus's mailing list posts, I worry there may be an indexing-breaking gloda bug happening.  Logs should hopefully make that more obvious.

In terms of tricks to make gloda re-index a folder, I think you want to use the Folder "Compact" option.  That does not throw away message meta-data and gloda has a specialized handler for compaction that should make the right things happen and fast.  But if gloda broke, nothing is going to fix indexing other than restarting Thunderbird.

Comment 6

[Wayne Mery (:wsmwk)]

klaus, does compact force the reindex?

Comment 7

[klaus]

52.7.0: neither repair nor compact force reindex. (testaccount with 30 messages only).
Gloda hangs on indexing sent folder (activity monitor), after deleting the global.sqlite to force reindexing. Now over 2 days and 10 restarts of TB.

If I tag a new message, it never gets a gloda_id (using glodaquilla).
I have many messages with gloda_id 2 (I don't remember exactly but think that is something like: cannot index)

If I search for tags in addon:  let query = Gloda.newQuery(Gloda.NOUN_MESSAGE);
  let tagArray = MailServices.tags.getAllTags({});  query.tags(...tagArray);

the collection never holds the newly tagged message (which doesn't  get gloda_id set, so maybe to be expected).

Error console reports an error in indexer.js:897:

code line 897 is: this.activeIterator.return() (error console: this.activeIterator.return is not a function)
in:  pop: function gloda_index_callbackhandle_pop

maybe this helps