479008 - gloda event-driven indexing should not interfere with the source of events

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

As discussed starting at bug 470329#c9, gloda right now has an annoying tendency to initiate event-driven processing in the midst of the flurry of activity causing the event.  This can do bad things to user responsiveness.  Particularly when gloda is at rest and is only dealing with occasional new events, it should defer indexing until no new events have been seen for a bit.

Comment 1

[Mark Surman]

Andrew: how difficult is this? Is it the kind of thing that one of Dave Humphrey's students at Seneca could work on? If so, maybe flag as 'student-project'.

Comment 2

[David Ascher (:davida)]

Naively: wouldn't judicious use of a setTimeout on the gloda event handler solve this with a few lines?

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

As davida says, the code required is a few lines of code; the unit testing required will be more sizable, but still fairly small.  The main thing is that it has to happen in the next two weeks.  I expect this is the wrong scope and time granularity.  Next gloda triage pass I'll try and mark things that I believe are suitable.

Comment 4

[Mark Surman]

Cool on looking for future student projects here. Just want to see Gloda succeed ... and niavely trying to point student energy at it :).

Comment 5

[John Beranek]

Is this what's causing an issue I discovered when trying out a Thunderbird 3 nightly at home?

Running on Vista, connecting to a local IMAP server (Dovecot), Thunderbird's UI is quite unresponsive when Thunderbird is started up.

My profile has a number of IMAP accounts in it, each with quite a few folders and 1000s of emails in a number of folders.

It seems that in this responsive time the search index is being built. Once the indexing has finished (again this is somewhat guesswork) the UI becomes responsive again...

Comment 6

[John Beranek]

(In reply to comment #5)
> It seems that in this responsive time the search index is being built. Once the
> indexing has finished (again this is somewhat guesswork) the UI becomes
> responsive again...

...in this _un_responsive time...

*sigh*

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

Gloda is currently not turned on by default in the nightlies.  Unless you've explicitly checked the "Enable Global Search and Indexer" in Preferences|Advanced|General, what you're seeing can't be this bug.

Comment 8

[John Beranek]

(In reply to comment #7)
> Gloda is currently not turned on by default in the nightlies.  Unless you've
> explicitly checked the "Enable Global Search and Indexer" in
> Preferences|Advanced|General, what you're seeing can't be this bug.

I have enabled this option.

Comment 9

[John Beranek]

When the UI is sluggish global-messages-db.sqlite is growing in size.

If I turn off the indexer and restart the UI returns to being responsive.

Comment 10

[Mark Surman]

>If I turn off the indexer and restart the UI 
>returns to being responsive.

I had this few experience a few weeks back. Stop using Gloda for now as a result.

Comment 11

[Andrew Sutherland [:asuth] (he/him)]

John, the problem you describe is not this bug; your problem sounds more along the lines of bug 465009 (now closed because believed addressed by bug 470329).

Please log a new bug.  Also, please make sure you have disabled Thunderbird's indexing/search support if you have enabled it.  On that bug, please:

Turn on gloda's dump" debug logging (
https://wiki.mozilla.org/Thunderbird:Using_Gloda ), run thunderbird from a console, and see what the debug output is correlated with in terms of the UI become most unresponsive.

I am particularly interested in lines of the following:
2009-04-08 13:10:49	gloda.indexer	DEBUG	Entering folder: imap://blah blah blah blah.

While gloda throttles itself, it is unable to throttle the foldering parsing, which can potentially cause serious UI issues when the folders have a lot of messages in them.

Also interesting are lines like:
2009-04-08 13:10:48	gloda.indexer	DEBUG	PERFORMANCE none average: 0 interval: 50 tokens: 5
2009-04-08 13:10:49	gloda.indexer	DEBUG	PERFORMANCE up average: 0.15900990092084347 interval: 40 tokens: 5

If things are performing badly, but gloda keeps saying "PERFORMANCE up", there will be trouble.  But again, a new bug please, as this bug is for a different set of issues and I will get confused.  Feel free to add me to the cc (put ":asuth") to ensure my attention is drawn from the get go.

Comment 12

[Andrew Sutherland [:asuth] (he/him)]

er, when I said "Thunderbird's indexing/search support" I meant to say "Thunderbird's vista indexing/search support" which is not gloda.  It is controlled by the "mail.winsearch.enable" preference.

Comment 13

[John Beranek]

(In reply to comment #11)
> Turn on gloda's dump" debug logging (
> https://wiki.mozilla.org/Thunderbird:Using_Gloda ), run thunderbird from a
> console, and see what the debug output is correlated with in terms of the UI
> become most unresponsive.

Hmm, I'm confused by these instructions. Set all the options, and then opened a Vista command prompt and typed "thunderbird". I was immediately returned to the command prompt, with Thunderbird seemingly running in the background. No debug comes out to the command prompt...

Should prolly take this to the new (not yet created) bug though I guess.

Comment 14

[Wayne Mery (:wsmwk)]

assuming this is the right bug (if not, kick me) ...

with indexing on and none of the whittler inducing extensions, message moves are sluggish for filtered and manual moves. annoyingly so for manual. for example move 100-200 bugmails from imap to local cause short groups of indexing (seen in activity monitor and status), and the moves are slow

perhaps unrelated, if I move all (or many) messages from a folder, sometimes one or two bogus messages are in the folder with the funky 12/13/1969 date

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to comment #14)
> ...
> are sluggish for filtered and manual moves. annoyingly so for manual. 

... to the point where I have turned off indexing for all my machines.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

I am pushing this out because we won't block beta 3 for this since we aren't pref-ing gloda on by default.  It would be a nice fix for beta 3 if the window doesn't close, though.

Comment 17

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Have a separate somewhat larger delay when we initiate indexing v1

Bug 465353 (rkent's more adaptive indexer) resolves a lot of the badness where gloda would interfere with normal processes.

This patch just increases the delay (via a new/separate constant) from when we want to start indexing to when we actually start indexing.  The goal is that when gloda is in steady-state and only doing event-driven indexing (which it can keep up with), that we defer our event-driven processing slightly to reduce interference/contention.

I have chosen 200ms arbitrarily as a delay that still allows gloda to be responsive while getting out of the way of any synchronous behavior.  In the event there is still stuff happening, the improved adaptive logic should keep us from interfering too much.

Comment 18

[David :Bienvenu]

Comment on attachment 397711 [details] [diff] [review]
Have a separate somewhat larger delay when we initiate indexing v1

looks reasonable to me.

Comment 19

[Mark Banner (:standard8)]

Checked in: http://hg.mozilla.org/comm-central/rev/9ba3c9960761