1095988 - Global indexing (Gloda) does not automatically re-index IMAP folders if messages not on disk ("Keep messages for this account on this computer" disabled) after the global-messages-db.sqlite file is deleted

Status: UNCONFIRMED

(Thunderbird :: Search, defect)

Description

[Ken Alker]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:33.0) Gecko/20100101 Firefox/33.0
Build ID: 20141023194920

Steps to reproduce:

1) Disable Global Search and Indexer in TB
2) Quit TB
2) Delete global-messages-db.sqlite
3) Launch TB
4) Enable Global Search and Indexer


Actual results:

global-messages-db.sqlite file is recreated and (*only*) INBOX is automatically re-indexed.


Expected results:

global-messages-db.sqlite file should be recreated and ALL mailboxes should be automatically re-indexed.

NOTES:
1) My mailboxes are all IMAP.
2) I have deselected "Keep messages for this account on this computer".
3) Disk Space is set to "synchronize the most recent 30 days".
4) When I click on a mailbox in the mailbox list, the mailbox headers are downloaded and the indexer runs (per Activity Manager, which is flaky so I'll open another bug), global-messages-db.journal pops up for a second, global-messages-db.sqlite increases in size, and the mailbox that was indexed is now successfully searchable.
5) My understanding from hours of reading is that the Global Indexer should work just fine even when using "offline folders" (which I interpret to mean no synchronizing or storying of IMAP data locally is being performed).

Comment 1

[Ken Alker]

I failed to mention in the "Steps to reproduce" section above that I deleted the entire Imap mail cache in TB's profile directory before re-launching TB.  I had deselected "Keep messages for this account on this computer" in order to free up some hard disk space, and in so doing, removed the Imap cache at the same time.

Comment 2

[Wayne Mery (:wsmwk)]

(In reply to Ken Alker from comment #0)
> User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:33.0)
> Gecko/20100101 Firefox/33.0
> Build ID: 20141023194920
> 
> Steps to reproduce:
> 
> 1) Disable Global Search and Indexer in TB
> 2) Quit TB
> 2) Delete global-messages-db.sqlite
> 3) Launch TB
> 4) Enable Global Search and Indexer

did you restart Thunderbird after step 4?


> Actual results:
> 
> global-messages-db.sqlite file is recreated and (*only*) INBOX is
> automatically re-indexed.
> 
> 
> Expected results:
> 
> global-messages-db.sqlite file should be recreated and ALL mailboxes should
> be automatically re-indexed.
> 
> NOTES:
> 1) My mailboxes are all IMAP.
> 2) I have deselected "Keep messages for this account on this computer".
> 3) Disk Space is set to "synchronize the most recent 30 days".
are you sure you got #3?
install glodaquilla addon and add the "on disk" column to show you which messages have been stored locally?

> 4) When I click on a mailbox in the mailbox list, the mailbox headers are
> downloaded and the indexer runs (per Activity Manager, which is flaky so
> I'll open another bug), global-messages-db.journal pops up for a second,
> global-messages-db.sqlite increases in size, and the mailbox that was
> indexed is now successfully searchable.
> 5) My understanding from hours of reading is that the Global Indexer should
> work just fine even when using "offline folders" (which I interpret to mean
> no synchronizing or storying of IMAP data locally is being performed).

re: #5, yes, in a limited fasion. i.e. to the extent of what message bodies and headers are stored locally

Comment 3

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #2)
>> ...
> > 5) My understanding from hours of reading is that the Global Indexer should
> > work just fine even when using "offline folders" (which I interpret to mean
> > no synchronizing or storying of IMAP data locally is being performed).
> 
> re: #5, yes, in a limited fasion. i.e. to the extent of what message bodies
> and headers are stored locally

To clarify, "offline folders" these days does mean data that has been downloaded. If you install the glodaquilla addon you can add an "on disk" column that shows the status.  

However, even if a message is not on disk, I believe it's headers should be indexed.

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to Ken Alker from comment #0)
> Steps to reproduce:
> 1) Disable Global Search and Indexer in TB
> 2) Quit TB
> 2) Delete global-messages-db.sqlite
> 3) Launch TB
> 4) Enable Global Search and Indexer
(In reply to Ken Alker from comment #1)
> I failed to mention in the "Steps to reproduce" section above that I deleted the entire Imap mail cache in TB's profile directory
> before re-launching TB. 
> I had deselected "Keep messages for this account on this computer" in order to free up some hard disk space, and in so doing, 
> removed the Imap cache at the same time.

Global Indexer can index message only when Global Inboxer can know that mail exists in a folder.
No one automatically access IMAP folder, unless ;
- IMAP folder is explcitly opened at folder pane (message header is feched upon folder open)
- Inbox : "Check new messges at/every" is checked.
- Other than Inbox : Option of "When Inbox is checked, check new messages of this folder too" is requested(Folder Properties)
- Any folder : auto-sync=enabled, imap folder is Offline-Use=On folder, then mails are auto-sync'ed
     "deselecting Keep messages for this account on this computer" == set Offline-Use=Off of all folder of this imap account.
     "re-selecting Keep messages for this account on this computer" == set Offline-Use=On  of all folder of this imap account.

Because you deleted all .msf files in which message header data is saved, no one can know aout messages in imap Mbox until one of above occurs.
In your case, on other than Inbox, any of above doesn't look to occur by your operations.

Comment 5

[Adam]

This is happening to me too. In my case, I right-clicked the properties of one of my IMAP folders and "Include in the global search results" was checked, as was "Select this folder for offline use". That should certainly be sufficient for messages in this folder to be included in the search results, but they aren't.

When I deleted and rebuilt my global database, the activity manager first rebuilt all my local folders, then it started rebuilding one of my IMAP folders. But it only got partway finished, saying it had indexed 36 messages, when the folder contained over 746 messages. Maybe something is causing it to abort the building of the database?

Comment 6

[Adam]

I found this page which seems to confirm that messages are only intended to be searchable if they're offline:

https://support.mozilla.org/en-US/kb/imap-synchronization

The problem I mention for me seems to be a separate problem that I seem to have solved. Somehow some of my messages weren't being stored for offline use if I moved them via the webmail interface outside Thunderbird, even though offline use was selected. This was causing the search to not find them. When I created a new Thunderbird profile and entered my account info from scratch, the search was then able to find messages in my IMAP folder. So I guess my Thunderbird profile had gotten corrupted somehow, probably from another bug and from being copied around over the years.