Some messages aren't indexed by gloda

RESOLVED INVALID

Status

MailNews Core
Database
--
major
RESOLVED INVALID
6 years ago
6 years ago

People

(Reporter: Usul, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
I've set up this new profile with two accounts.

Activity manager is not telling me that messages are being indexed (ergo both accounts have been indexed by now).

I wanted to search for banner to see standard8 emails I know i have on that account (as it's being subscribed to the mozmill mailing list). Results are empty. I tried to search for mark banner, and get prompted for auto completion to another mark that I got email on that account. Mark banner doesn't show up any results either.

Using the QFB I filtered for bann and get 11 messages.

I can save the gloda db and msf files for that account, I can even share the account credentials (it's a gmail one). More importantly how do we figure out why some messages aren't indexed ? what logging, debugging flags can I set to figure out what is going on ?
(Reporter)

Comment 1

6 years ago
sorry for the spam , indexing was still going on - but not visible in the activity manager
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → INVALID

Comment 2

6 years ago
So isn't there a problem why it is not shown?

This happened to me too on a clean new profile indexing all messages on IMAP. It couldn't find messages that must have been there.
http://mxr.mozilla.org/comm-central/source/mail/components/activity/modules/glodaIndexer.js#211

(in onJobCompleted:)
    // We only create activity events when specific folders get indexed,
    // since event-driven indexing jobs are too numerous.  We also only create
    // them when we ended up indexing something in the folder, since otherwise
    // we'd spam the activity manager with too many "indexed 0 messages" items
    // that aren't useful enough to justify their presence in the manager.
    // TODO: Aggregate event-driven indexing jobs into batches significant
    // enough for us to create activity events for them.

Presumably the indexing Ludovic is experiencing is being driven by auto-sync downloading the messages bit by bit in the background and event-driven indexing processing things.  This should result in the activity manager showing something while the event-driven indexing is going on, but it won't leave a historical item behind when it finishes.  The rationale is that the log would be full of: (gloda indexed 1 item) (gloda indexed 1 item) (gloda indexed 2 items).  It's conceivable that gloda is fast enough on fast hardware that the items don't stay up there very long.

Since auto-sync operates on a folder-by-folder basis and so the event-driven indexing will effectively be operating out of a small number of folders at a time, there's definitely an argument for having gloda's activity monitor go to a little more work to try and leave useful historical blobs.  It could also just update a blob that shows a tally for event-driven indexing.  That should likely be marked as an enhancement request with "[gloda key]" in the whiteboard.
Also, ludo, you probably want to be using glodaquilla which adds that extra column so you can see if a message has been indexed by gloda or not.  You may need to force its compatibility a bit (or try and bribe rkent :)
(Reporter)

Comment 5

6 years ago
(In reply to Andrew Sutherland (:asuth) from comment #4)
> Also, ludo, you probably want to be using glodaquilla which adds that extra
> column so you can see if a message has been indexed by gloda or not.  You
> may need to force its compatibility a bit (or try and bribe rkent :)

Yeah that was my way to go. but wanted to leave a bit more time. I just thought that not seeing any indexing activity meant indexing was done.
You need to log in before you can comment on or make changes to this bug.