1247240 - Enable configurable time sync window to mitigate performance problems on very large calendars

Status: NEW

(Calendar :: Provider: CalDAV, defect)

Description

[André Bação]

Attachment: lightning-4.0.5.2-sm+tb-linux.patch

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.28 Safari/537.36

Steps to reproduce:

Have a calDav server serving a calendar with 10.000 events.
When syncing, Lightning downloads all the events and shows them.
Tested in Oracle Communications Calendar Server and Zimbra.



Actual results:

This makes Lightning and Thunderbird VERY SLOW.
When Lightning is disabled, the problem isn't there.
There isn't a problem with network speeds, the problem happens when a user has many events to sync over calDav.

links related:
https://bugzilla.mozilla.org/show_bug.cgi?id=733039




Expected results:

Possibility to choose in Lightning a sync time window.
The attached patch implements this feature by using a pref that controls the number of months that Lightning will sync ahead and behind current date.
The default value is 60 Months = 5 years.

Comment 1

[Wayne Mery (:wsmwk)]

There is also 
bug 1224249 (which has a WIP patch)
Bug 848024
bug 530423

Comment 2

[Stefan Sitter [:ssitter]]

As far as I known Lightning will only request the data it is configured to display. If you select a filter like 'All events' than it will request all data. But if you set the filter to e.g. 'current month' than it should request only this data.

Comment 3

[André Bação]

@Stefan Sitter: Lightning request all data, even the one you didn't configured to be displayed.

@Wayne Mery: 
bug 1224249: Tested offline support option and it was slower than without, but it is unrelated.
The other bugs are quite old but yes, the problem seems to be the same.

Comment 4

[Dominik Wnęk]

I can confirm what Stefan said about selecting filters having an effect on Lightning performance, especially on startup. We're all very careful to make sure to never close Thunderbird with "all events" selected - as it then takes something like 30 minutes to open it back again and get it to a usable state, with multiple stalls and "script is not responding" errors. If something like "next 31 days" is selected the start up and sync take maybe 5 minutes. This is with 13 CalDAV calendars, each with a few hundred events in it.

Anyway, I'd really _love_ to be able to select the synced time window, like this patch does. It should really help with performance (Thunderbird with Lightning is the most important tool we use in out business). Is there a chance it can get integrated into Lightning soon? Otherwise, how would I go about patching Lightning myself?

Thanks,

Comment 5

[André Bação]

Attachment: Patch for user defined time range sync [final]

This patch when applied have the following behaviour:

- if non is changed, thinderbird and lightning keep their original behaviour
- if the time range window is defined, lighting will only sync the events inside user defined time range.

Comment 6

[[:MakeMyDay]]

André, thanks for digging into this and providing a patch. New contributors are always welcome, although there are some obstacles to overcome  at the beginning (see below)!

Some drive by feedback/questions:

Have you tested the behaviour of recurring events that start before the replicated time range and have occurences within? While recurring events are knwon to be a main driver for slowing down the process, we must be sure they are not excluded when syncing. Not sure how the server deals with that.

What happens if the user has defined a time range (say a narrow one because he/she has many big calendars) and moves back or forward to a period outside?

Apart of that, a patch needs to follow some conventions [1][2][3] to be ready for review/checkin and more important must be based on the (latest version of the) repository code and not the packaged release version of Lightning. Unfortunately, these have different folder structures and file locations even within calendar and also other commits might have been applied since release.

Your patch is prepared against the packaged version of Lightning. It's a little extra effort, but it's worth it to make your patch go into the code base and I hope you don't get distracted - otherwise another developer would have to transform your patch into applicable code first before it's even considered, which is not likely to happen due to our limited ressources.

Finally, if you have prepared and uploaded a patch, make sure to ask for review or feedback (you can do so in the details view of the respective attachment by setting r? or f? flags; for r? you're also prompted with a suggested reviewer) - otherwise your patch might rot here forever without being noticed by anyone, which is probably not what you want ;-)


[1] https://developer.mozilla.org/en-US/docs/Mercurial/Using_Mercurial
[2] https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Thunderbird_build (only if you want to build yourself with your changeset applied; if you encouter problems while doing so, the easiest would be to ask in #maildev on IRC)
[3] https://wiki.mozilla.org/Calendar:Style_Guide

Comment 7

[André Bação]

Hi

Recurring events: are not affected. They still are synched even if a time window is selected.

If a time window is small, the sync will only occur in that window. The user can normally add past or future events but when a sync occurs, that event will disappear from Thunderbird but will stay in the server.

But all this is easily tested with the patch ;) 

I will check [1] [2] and [3] and try to fulfil your requirements 

(In reply to MakeMyDay from comment #6)
> André, thanks for digging into this and providing a patch. New contributors
> are always welcome, although there are some obstacles to overcome  at the
> beginning (see below)!
> 
> Some drive by feedback/questions:
> 
> Have you tested the behaviour of recurring events that start before the
> replicated time range and have occurences within? While recurring events are
> knwon to be a main driver for slowing down the process, we must be sure they
> are not excluded when syncing. Not sure how the server deals with that.
> 
> What happens if the user has defined a time range (say a narrow one because
> he/she has many big calendars) and moves back or forward to a period outside?
> 
> Apart of that, a patch needs to follow some conventions [1][2][3] to be
> ready for review/checkin and more important must be based on the (latest
> version of the) repository code and not the packaged release version of
> Lightning. Unfortunately, these have different folder structures and file
> locations even within calendar and also other commits might have been
> applied since release.
> 
> Your patch is prepared against the packaged version of Lightning. It's a
> little extra effort, but it's worth it to make your patch go into the code
> base and I hope you don't get distracted - otherwise another developer would
> have to transform your patch into applicable code first before it's even
> considered, which is not likely to happen due to our limited ressources.
> 
> Finally, if you have prepared and uploaded a patch, make sure to ask for
> review or feedback (you can do so in the details view of the respective
> attachment by setting r? or f? flags; for r? you're also prompted with a
> suggested reviewer) - otherwise your patch might rot here forever without
> being noticed by anyone, which is probably not what you want ;-)
> 
> 
> [1] https://developer.mozilla.org/en-US/docs/Mercurial/Using_Mercurial
> [2]
> https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/
> Build_Instructions/Simple_Thunderbird_build (only if you want to build
> yourself with your changeset applied; if you encouter problems while doing
> so, the easiest would be to ask in #maildev on IRC)
> [3] https://wiki.mozilla.org/Calendar:Style_Guide

Comment 8

[[:MakeMyDay]]

(In reply to André Bação from comment #7)
> Recurring events: are not affected. They still are synched even if a time
> window is selected.

So all recurring events are synced as long as any of its occurrences is within the the selected time range, right? This is what I expected, but I wasn't sure.

> If a time window is small, the sync will only occur in that window. The user
> can normally add past or future events but when a sync occurs, that event
> will disappear from Thunderbird but will stay in the server.

That's what I was afraid of when asking. This will lead to an odd user expirience (especially if offline support is disabled or you want to view your calendar outside the time range) and end up in variety of user complaints and bug reports mentioning Lightning to loose their data.

To resolve this, it might be appropriate to implement either a trigger from UI interaction to load further time ranges (in the background) if needed or to start syncing only the current period (maybe depending on the current calendar view setting) for each calendar in a first place and then incrementally load all of the remaining data (without time limit).

> But all this is easily tested with the patch ;) 

It would, once it's a patch against comm-central (which is the relevant repository here, btw) ;-)

> I will check [1] [2] and [3] and try to fulfil your requirements

This is much appreciated, thanks again!

Comment 9

[Philipp Kewisch [:Fallen][📱📆]]

There is some work underway from Linagora to provide an alternative caldav provider that allows time-range querying when cache mode is off. As far as I know it doesn't actually fix the performance issues completely though, likely given Lightning's architecture could use some changes.

It is not actually as easy as the patch you put here, there are a lot of edge cases to cover. I think the best way forward would not actually be limiting the caldav provider in funtionality, but working on Lightning's other components to make them fast.

A few things I'd be interested in:
* an indexedDB-based storage provider that uses as little xpcom as possible.
* alternatively finishing off bug 501689
* make Lightning's backend more promise based (and give a sufficient amount of pauses on the event loop for UI events to happen)
* Move some processing into workers (probably requires use of ical.js since we don't have xpcom in workers)

That said, if you are interesting in improving Lightning's performance, I am happy to mentor you into fixing the right bugs to improve performance. Would you be up for that?

Comment 10

[Richard Leger]

FYI, I have created this bug that may be partially related Bug 1544375 "Ligtning - Add network calendar UI options: Chunk size, Initial sync time range filters (sync timespan future and past)"

Comment 11

[Wayne Mery (:wsmwk)]

(André seems to be gone)

Comment 12

[André Bação]

I'm here but this wasn't on my radar anymore.
This is was made to solve a specific problem, not to create a new feature (I will leave that to the product owner).

Comment 13

[Wayne Mery (:wsmwk)]

André, is it better for you now with version 128?