461032 - Integrate unifinder search into Gloda and Thunderbird Search Bar

Status: NEW

(Calendar :: General, enhancement)

Description

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

As soon as exptoolbar hits the nightlys, we need to adapt calendar to use that search.

I haven't fully understood what is needed, but iiuc then I need to implement a nsIAutocompleteSearch(Result) interface around the unifinder search.

We need to find out how the search is best done. I think its unreasonable to search through all events, but since the toolbar has no contraints by default, this would be the logical thing to do :-(

We might want to create a new interface for providers to allow searching for events. The storage provider could add a fulltext search on certain fields, to return the events faster.

Other providers could easily implement this, there may be some restrictions (i.e google can only do a fulltext search over all fields), but I think this would speed up our search a lot.

I also looked into intermediately moving the search into the existing search toolbaritem, but its probably not worth the effort.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

How do the current providers implement their search?  Ideally, they would not be clever/efficient about things, which allows us to easily implement things using gloda's framework without duplication.

For now, the idea is that "gloda is an index".  It is not an authoritative data store; just an index.

My imagined implementation would be this:
1) We create a "simple" javascript class for calendar events.  It has/is a unique identifier.  It knows the start and stop times of an event and the 'location' of the actual underlying event[A].  Each instance represents a row in a new table in the gloda SQLite database.  (Other information gets to be attributes.)  When defining the schema, we take advantage of gloda's (thin) support to create a related full-text table that indexes the title of the event as well as any comments/the like.

2) We create a javascript class for task events.  It knows the name of the task and the 'location' of the underlying task.  (Other information gets to be attributes.)  Again, full-text table too.

3) We create an indexer for calendar items.  It walks the calendars, creating the gloda representations above and putting any interesting text into the fulltext for searching purposes.  Here is we store the fun stuff, like other people involved, tags, etc. as attributes.  (Basically, anything that we could have more than one of needs to be an attribute, if it happens at most once and is pretty likely to be present, it can go on the row.)  The indexer would also be event-driven, but needs to know how to walk things too.

4) We create an indexer for tasks.  Same deal; it stashes the fulltext and gets to make attributes out of tags applied to the task, etc.

maybe 5)  If thunderbar/gloda hasn't gotten around already to providing noun detection relating to time, we add that.  This may just entail stealing/unifying code from ubiquity.  (ThunderBar's desired functionality is arguably a subset of Ubiquity's, and we need to try and conceptually unify them at some point.  In the future, not now.)

6) We create a completion-helper and query-helper for thunderbar, or generalize the existing completion helpers be able to do this automatically with minimal definition by the lightning/calendar-code.  Ideally, there should be little completion code that is specific to lightning; time, people, and tags should already be dealt with.  This leaves text searching, which I'm not sure we want to complete on (this is currently an open question with messages as well.)  The query helper needs to take the time/tags/people/what not and convert this into queries against the lightning gloda stores.  Again, this might happen automatically.

7) We create XBL bindings to present calendar items and tasks to the user, at least for a combined results page.  This may just mean adapting existing lightning XBL to pierce the gloda representations into the actual lightning representation.  This may also mean creating a new results tab that just shows the existing calendars with matching items highlighted or something.  That's really a clarkbw question.  The point is that the result presentation is a real issue, especially when we're also showing messages and people.

==== Important notes/issues:

A) The data model for events I propose is obviously simplistic, and does not account for capturing recurring events as such.  I propose that we flatten recurring events to the database for some arbitrary time period into the future, which we dub the "event horizon" (for obvious nerd appeal reasons).  Although we could graft some form of complicated dynamic extrapolation for queries, I think that's probably a waste of time.

I assert that any period of time a user needs to care about can be persisted to the database.  If some user needs to know how his weekly status call impacts his life 10 years in the future, he can pay the cost of having 10 years worth of those events hit the database.  If the user issues a gloda query that crosses the 'event horizon', we provide them with the option of moving the event horizon up to/beyond the range of their query.  (This is not to say that Calendar would stop letting people browse into the deep future for amusement; we just wouldn't index it.)

==== Dealing with reality

I don't know how Calendar works currently.  What I do know is that we need to deal with Calendar's lowest common denominator (LCD).  If one of your implementations is super fancy and databased-back, but there's one that just naively reads an .ics file from disk every time you want to do something, our reality is the naive one.  In the naive case, my proposal above is great.  If the LCD is really smart already then I'll need to re-think things, although I'm not sure it change things greatly.  It's probably better do duplicate some data on-disk/in-memory than have excessively complex code bridging gloda to calendar.

Comment 2

[Pete Riley]

I would prefer to keep it simple and search through all events of all enabled calendars without regard to start/end dates, at least with ICS calendars.  Consider that the entire ics calendars are probably already loaded in memory and the average user probably doesn't have more than 1 MB of ics data to search through (i.e. very fast on a modern computer; IMO no need to bother with indexes for ics).

I think that any limitations of the CalDAV/Google/WCAP providers (e.g. if they don't load the entire calendars into memory) would make them the least common denominator. :)  I hope that their "limitations" do not force date restrictions when we search our ics calendars.

Comment 6

[Thomas D. (:thomas8)]

Notwithstanding the goodness of this plan, it's also very complex. Maybe we could start with something more lightweight and much easier to implement, like bug 316916 - Filter for Tasks. Which would also add a lot of value, since we currently have no way of searching/filtering tasks.