482724 - Thunderbird should re-index mail for OS search tools if mozmsgs folders are removed

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[David Humphrey (:humph)]

Spin-off from bug 290057 comment 181 -- if "cleaning" tools or the user remove their search metadata (e.g., ~/Library/Caches/Metadata) we should re-index all mail.  Perhaps this means searching for the presence of the mozmsg dirs on app startup?

Comment 1

[Rain [:rain1] (inactive)]

Attachment: patch, v1

Whew, this bug was a lot more trouble than I thought!

So my initial, naive solution was to store a boolean state storing whether this folder is being reindexed. However, I realized that the state wouldn't persist across a shutdown and startup, and didn't really want to add an additional pref for something so ephemeral, so a bit of reworking about how indexing status is stored was required.

This solution uses a "reindex time", which would be set globally, and per-folder as necessary. If any of the two times are later than the time stored on the message header, we check if the file exists. If it does, then we update the reindex time, otherwise we stream and index the file.

Now the fun part begins -- the old _findNextFolderToIndex would run through all folders again and again, looking for a non-existent folder. However, there can exist folders with no messages, so earlier those folders wouldn't be created at all. The solution to this was to create the folder as soon as we encountered it.

That wasn't all, however. If we shutdown in the middle of indexing the folder, we'd not reindex the remaining messages at next startup! Also, updating the lastFolderIndexedUri was "wrong", in the sense that we'd be looking at lots more folders that we know we've indexed.

The final approach here is to use a generator function, so that we don't run through the folders again and again. We also have control over the last indexed folder here, so that we update it if it comes after what we know as the last indexed folder, but not after a missing folder.

The advantage of this approach is that it should be fairly easy to add a reindex all command after this -- just delete the mozmsgs folders and bump up the global reindex time.

Comment 2

[Mitra Ardron]

I've been looking at the spotlight indexing, one key problem is that it doesn't reindex when Spotlight is first turned on, it only indexes messages that are opened themselves, which in effect means that spotlight appears not to work for people just starting to use it, i.e. not just in error cases.

Comment 3

[David :Bienvenu]

ok, building this now...

Comment 4

[David :Bienvenu]

Comment on attachment 368239 [details] [diff] [review]
patch, v1

thx, Sid.

Comment 5

[Mark Banner (:standard8)]

Checked in: http://hg.mozilla.org/comm-central/rev/906a492a1a49

Comment 6

[Rain [:rain1] (inactive)]

Mitra -- can you now see existing messages being indexed? Note that you have to leave your computer idle for around 30 seconds before existing messages start getting indexed.

Comment 7

[Mitra Ardron]

No - it doesn't appear to be reindexing folders that contain old messages, not unless its REALLY slow at doing it, I don't seem to have a way to tell whether its reindexing other than to look at the dates on folders in ~/Library/Caches/Metadata/Thunderbird but looking there I'm not seeing new stuff other than folders that mail is coming into.

- Mitra

Comment 8

[Mitra Ardron]

Actually, it looks like it is indexing things, its just taking a long time to get through the backlog of messages, which I think is ok since that only happens when this feature is first turned on.

Comment 9

[Chris Denesha]

I don't think older messages ever get rebuilt. I clear my personal caches every month or two and only get mozmsgs for new emails that come in.
chris