1224249 - Slow startup with CalDav when cache (offline support) enabled because all events are loaded from cache

Status: NEW

(Calendar :: Provider: CalDAV, defect)

Description

[Raffael]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0 Iceweasel/38.2.1
Build ID: 20150828090403

Steps to reproduce:

When using caldav calendars with offline support and a lot of events, the startup time of Thunderbird can get really long (several minutes in my case). The time is consumed before any UI gets visible. 

I did some testing and could narrow dawn the problem to one function in the startup procedure. I'm not a familiar with the lightning code so I might not be able to present a patch fast, but maybe someone knowing the code might improve the situation quickly with the following information.

The problem is in this function: 
ensureMetaData() in calDavCalendar.js

The method is called at the end of fetchCachedMetaData() which loads all metadata from the sqlite table cal_metadata into mItemInfoCache.
ensureMetaData now loads all Events from the cache and checks if there is metadata in mItemInfoCache for every event. Loading all the events takes a lot of time. For a single calendar with 5500 entries it takes 25 seconds on my computer. 
IMHO ensureMetaData should only load events returned by a query like this: 
SELECT * FROM cal_events WHERE id NOT IN (SELECT item_id FROM cal_metadata)

I hope this will help to further improve a really nice Product (Thunderbird with lightning)

Comment 1

[Raffael]

Attachment: ensureMeta.patch

I did some hacking and created a patch. Please keep in mind, that a have no experience in coding thunderbird add-ons. This patch is probably not valid for every case and does not follow any mozilla coding conventions.
It is more a proof of concept. Startup time with my test calendar changed from more than 30 seconds to about 10, when this patch is applied.

Comment 2

[Wayne Mery (:wsmwk)]

dupe of bug 848024 or bug 530423 (which wants bug 462277)

Comment 3

[Raffael]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #2)
> dupe of bug 848024 or bug 530423 (which wants bug 462277)

Why do you think this is a duplicate of one of these bugs?
I don't think it is.

bug 848024 is about the initial sync of a calendar (adding the calendar). Which IMHO is still not really fast but much better since WAL was enabled (Bug 918175). This bug is is about every start after the calendar was added.

bug 530423 is about large calendars in uncached mode, while the problem I reported in this bug is only in cached mode

Comment 4

[Wayne Mery (:wsmwk)]

(In reply to Raffael from comment #3)
> (In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #2)
> > dupe of bug 848024 or bug 530423 (which wants bug 462277)
> ....
> bug 530423 is about large calendars in uncached mode, while the problem I
> reported in this bug is only in cached mode

because of bug 462277.  But I'm no expert here. So take my comments with a grain of salt

Comment 5

[[:MakeMyDay]]

Matthew, can you take a look to what Raffael proposed as you also did the patch in bug 827078, which introduced ensureMetaData() ?

Comment 6

[Matthew Mecca [:mmecca]]

I think this approach makes sense. It would need an additional database statement to check for tasks without metadata, and also append those id's to the results of the getIdsWithoutMetaData function. We could probably also cut out the getItem overhead altogether in the ensureMetaData function, and just generate the locationPath metadata that we need from the item id's directly in the for loop.

Comment 7

[Philipp Kewisch [:Fallen][📱📆]]

(In reply to Matthew Mecca [:mmecca] from comment #6)
> We could probably also
> cut out the getItem overhead altogether in the ensureMetaData function, and
> just generate the locationPath metadata that we need from the item id's
> directly in the for loop.

Some servers don't use the item UID for the uri, so unfortunately we must keep locationPath.