Thunderbird should re-index mail for OS search tools if mozmsgs folders are removed

RESOLVED FIXED in Thunderbird 3.0b3

Status

RESOLVED FIXED
10 years ago
7 years ago

People

(Reporter: humph, Assigned: sid0)

Tracking

Trunk
Thunderbird 3.0b3

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

10 years ago
Spin-off from bug 290057 comment 181 -- if "cleaning" tools or the user remove their search metadata (e.g., ~/Library/Caches/Metadata) we should re-index all mail.  Perhaps this means searching for the presence of the mozmsg dirs on app startup?
(Assignee)

Comment 1

10 years ago
Created attachment 368239 [details] [diff] [review]
patch, v1

Whew, this bug was a lot more trouble than I thought!

So my initial, naive solution was to store a boolean state storing whether this folder is being reindexed. However, I realized that the state wouldn't persist across a shutdown and startup, and didn't really want to add an additional pref for something so ephemeral, so a bit of reworking about how indexing status is stored was required.

This solution uses a "reindex time", which would be set globally, and per-folder as necessary. If any of the two times are later than the time stored on the message header, we check if the file exists. If it does, then we update the reindex time, otherwise we stream and index the file.

Now the fun part begins -- the old _findNextFolderToIndex would run through all folders again and again, looking for a non-existent folder. However, there can exist folders with no messages, so earlier those folders wouldn't be created at all. The solution to this was to create the folder as soon as we encountered it.

That wasn't all, however. If we shutdown in the middle of indexing the folder, we'd not reindex the remaining messages at next startup! Also, updating the lastFolderIndexedUri was "wrong", in the sense that we'd be looking at lots more folders that we know we've indexed.

The final approach here is to use a generator function, so that we don't run through the folders again and again. We also have control over the last indexed folder here, so that we update it if it comes after what we know as the last indexed folder, but not after a missing folder.

The advantage of this approach is that it should be fairly easy to add a reindex all command after this -- just delete the mozmsgs folders and bump up the global reindex time.
Attachment #368239 - Flags: review?(bienvenu)

Comment 2

10 years ago
I've been looking at the spotlight indexing, one key problem is that it doesn't reindex when Spotlight is first turned on, it only indexes messages that are opened themselves, which in effect means that spotlight appears not to work for people just starting to use it, i.e. not just in error cases.

Comment 3

10 years ago
ok, building this now...

Updated

10 years ago
Attachment #368239 - Flags: review?(bienvenu) → review+

Comment 4

10 years ago
Comment on attachment 368239 [details] [diff] [review]
patch, v1

thx, Sid.
(Assignee)

Updated

10 years ago
Keywords: checkin-needed
Checked in: http://hg.mozilla.org/comm-central/rev/906a492a1a49
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 3.0b3
(Assignee)

Comment 6

10 years ago
Mitra -- can you now see existing messages being indexed? Note that you have to leave your computer idle for around 30 seconds before existing messages start getting indexed.

Comment 7

10 years ago
No - it doesn't appear to be reindexing folders that contain old messages, not unless its REALLY slow at doing it, I don't seem to have a way to tell whether its reindexing other than to look at the dates on folders in ~/Library/Caches/Metadata/Thunderbird but looking there I'm not seeing new stuff other than folders that mail is coming into.

- Mitra

Comment 8

10 years ago
Actually, it looks like it is indexing things, its just taking a long time to get through the backlog of messages, which I think is ok since that only happens when this feature is first turned on.

Comment 9

7 years ago
I don't think older messages ever get rebuilt. I clear my personal caches every month or two and only get mozmsgs for new emails that come in.
chris
You need to log in before you can comment on or make changes to this bug.