Closed Bug 479008 Opened 12 years ago Closed 11 years ago

gloda event-driven indexing should not interfere with the source of events

Categories

(MailNews Core :: Database, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
Thunderbird 3.0b4

People

(Reporter: asuth, Assigned: asuth)

References

(Blocks 1 open bug)

Details

(Keywords: perf, Whiteboard: [b4ux][gloda key][no l10n impact])

Attachments

(1 file)

As discussed starting at bug 470329#c9, gloda right now has an annoying tendency to initiate event-driven processing in the midst of the flurry of activity causing the event.  This can do bad things to user responsiveness.  Particularly when gloda is at rest and is only dealing with occasional new events, it should defer indexing until no new events have been seen for a bit.
Andrew: how difficult is this? Is it the kind of thing that one of Dave Humphrey's students at Seneca could work on? If so, maybe flag as 'student-project'.
Naively: wouldn't judicious use of a setTimeout on the gloda event handler solve this with a few lines?
As davida says, the code required is a few lines of code; the unit testing required will be more sizable, but still fairly small.  The main thing is that it has to happen in the next two weeks.  I expect this is the wrong scope and time granularity.  Next gloda triage pass I'll try and mark things that I believe are suitable.
Whiteboard: [b3ux][m4]
Cool on looking for future student projects here. Just want to see Gloda succeed ... and niavely trying to point student energy at it :).
Is this what's causing an issue I discovered when trying out a Thunderbird 3 nightly at home?

Running on Vista, connecting to a local IMAP server (Dovecot), Thunderbird's UI is quite unresponsive when Thunderbird is started up.

My profile has a number of IMAP accounts in it, each with quite a few folders and 1000s of emails in a number of folders.

It seems that in this responsive time the search index is being built. Once the indexing has finished (again this is somewhat guesswork) the UI becomes responsive again...
(In reply to comment #5)
> It seems that in this responsive time the search index is being built. Once the
> indexing has finished (again this is somewhat guesswork) the UI becomes
> responsive again...

...in this _un_responsive time...

*sigh*
Gloda is currently not turned on by default in the nightlies.  Unless you've explicitly checked the "Enable Global Search and Indexer" in Preferences|Advanced|General, what you're seeing can't be this bug.
(In reply to comment #7)
> Gloda is currently not turned on by default in the nightlies.  Unless you've
> explicitly checked the "Enable Global Search and Indexer" in
> Preferences|Advanced|General, what you're seeing can't be this bug.

I have enabled this option.
When the UI is sluggish global-messages-db.sqlite is growing in size.

If I turn off the indexer and restart the UI returns to being responsive.
>If I turn off the indexer and restart the UI 
>returns to being responsive.

I had this few experience a few weeks back. Stop using Gloda for now as a result.
John, the problem you describe is not this bug; your problem sounds more along the lines of bug 465009 (now closed because believed addressed by bug 470329).

Please log a new bug.  Also, please make sure you have disabled Thunderbird's indexing/search support if you have enabled it.  On that bug, please:

Turn on gloda's dump" debug logging (
https://wiki.mozilla.org/Thunderbird:Using_Gloda ), run thunderbird from a console, and see what the debug output is correlated with in terms of the UI become most unresponsive.

I am particularly interested in lines of the following:
2009-04-08 13:10:49	gloda.indexer	DEBUG	Entering folder: imap://blah blah blah blah.

While gloda throttles itself, it is unable to throttle the foldering parsing, which can potentially cause serious UI issues when the folders have a lot of messages in them.

Also interesting are lines like:
2009-04-08 13:10:48	gloda.indexer	DEBUG	PERFORMANCE none average: 0 interval: 50 tokens: 5
2009-04-08 13:10:49	gloda.indexer	DEBUG	PERFORMANCE up average: 0.15900990092084347 interval: 40 tokens: 5

If things are performing badly, but gloda keeps saying "PERFORMANCE up", there will be trouble.  But again, a new bug please, as this bug is for a different set of issues and I will get confused.  Feel free to add me to the cc (put ":asuth") to ensure my attention is drawn from the get go.
er, when I said "Thunderbird's indexing/search support" I meant to say "Thunderbird's vista indexing/search support" which is not gloda.  It is controlled by the "mail.winsearch.enable" preference.
(In reply to comment #11)
> Turn on gloda's dump" debug logging (
> https://wiki.mozilla.org/Thunderbird:Using_Gloda ), run thunderbird from a
> console, and see what the debug output is correlated with in terms of the UI
> become most unresponsive.

Hmm, I'm confused by these instructions. Set all the options, and then opened a Vista command prompt and typed "thunderbird". I was immediately returned to the command prompt, with Thunderbird seemingly running in the background. No debug comes out to the command prompt...

Should prolly take this to the new (not yet created) bug though I guess.
Blocks: 487375
Keywords: perf
Whiteboard: [b3ux][m4] → [b3ux][m6]
Status: ASSIGNED → NEW
Whiteboard: [b3ux][m6] → [b3ux][m6][gloda key]
Blocks: 496646
assuming this is the right bug (if not, kick me) ...

with indexing on and none of the whittler inducing extensions, message moves are sluggish for filtered and manual moves. annoyingly so for manual. for example move 100-200 bugmails from imap to local cause short groups of indexing (seen in activity monitor and status), and the moves are slow

perhaps unrelated, if I move all (or many) messages from a folder, sometimes one or two bogus messages are in the folder with the funky 12/13/1969 date
Flags: blocking-thunderbird3?
(In reply to comment #14)
> ...
> are sluggish for filtered and manual moves. annoyingly so for manual. 

... to the point where I have turned off indexing for all my machines.
I am pushing this out because we won't block beta 3 for this since we aren't pref-ing gloda on by default.  It would be a nice fix for beta 3 if the window doesn't close, though.
Whiteboard: [b3ux][m6][gloda key] → [b4ux][gloda key]
Target Milestone: Thunderbird 3.0b3 → Thunderbird 3.0b4
Flags: blocking-thunderbird3? → blocking-thunderbird3+
Whiteboard: [b4ux][gloda key] → [b4ux][gloda key][no l10n impact]
Bug 465353 (rkent's more adaptive indexer) resolves a lot of the badness where gloda would interfere with normal processes.

This patch just increases the delay (via a new/separate constant) from when we want to start indexing to when we actually start indexing.  The goal is that when gloda is in steady-state and only doing event-driven indexing (which it can keep up with), that we defer our event-driven processing slightly to reduce interference/contention.

I have chosen 200ms arbitrarily as a delay that still allows gloda to be responsive while getting out of the way of any synchronous behavior.  In the event there is still stuff happening, the improved adaptive logic should keep us from interfering too much.
Attachment #397711 - Flags: review?(bienvenu)
Comment on attachment 397711 [details] [diff] [review]
Have a separate somewhat larger delay when we initiate indexing v1

looks reasonable to me.
Attachment #397711 - Flags: review?(bienvenu) → review+
Keywords: checkin-needed
Whiteboard: [b4ux][gloda key][no l10n impact] → [b4ux][gloda key][no l10n impact][has reviewed patch]
Checked in: http://hg.mozilla.org/comm-central/rev/9ba3c9960761
Status: NEW → RESOLVED
Closed: 11 years ago
Keywords: checkin-needed, perf
Resolution: --- → FIXED
Whiteboard: [b4ux][gloda key][no l10n impact][has reviewed patch] → [b4ux][gloda key][no l10n impact]
Keywords: perf
You need to log in before you can comment on or make changes to this bug.