362987 - Large calendars with many recurring items slow down the app

Status: NEW

(Calendar :: Provider: Local Storage, defect)

Description

[Domingo Alvarez]

User-Agent:       Opera/9.01 (Windows NT 5.0; U; en)
Build Identifier: SunBird/0.41a

Whe populates with several recurrent events like using the http://ical.mac.com/rainbowfly/FLY-RADIO32SCHEDULE.ics it becomes unusable, too slow, I did some research and found that in the function getItems from local storage if we insert a this.mDB.beginTransaction() at the begining and a this.mDB.commitTransaction() at the end we can reduce the time spent to half but it is still to slow, when moving fron one day to another it seeks and proccess 200 recurrent records to show only 6 events in the calendar and other six in the top list filtered by today events.

It seems that the actual algorithm is not apropriate because it get worst when the number of recurrencies grow.

For comparison only I downloaded Personal calendar from www.calendarscope.com the shareware download and importesd the same calendar from Fly Radio and it's performance is astonishing, if we can solve this problem the claendar will be usable, because right know it's only a proof of concept, a good one but not usable.

Reproducible: Always

Steps to Reproduce:
1.import a calendar from http://ical.mac.com/rainbowfly/FLY-RADIO32SCHEDULE.ics
2.try to use it and navigate through it
3.comment a line on calStorageCalendar.js and see the difference, there is the slowdown
for each (var evitem in resultItems) {
///////   count += handleResultItem(evitem.item, evitem.flags, Ci.calIEvent);
   if (checkCount())
      return;
}


Actual Results:  
A lot slow, wait a bit, it works but take a long time to show and to move from day to day.

Expected Results:  
Navigate smootly. 

Navigate smootly.

Comment 1

[Damian Szczepanik]

I can confirm it with
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a2pre) Gecko/20070114 Calendar/0.6a1

but it also looks like duplicate of bug 355454

Comment 2

[cmtalbert]

Joey, you might want to take a look at this w.r.t. your other performance fixes.

Damian, we decided that we do not have enough information at this time as to whether or not 355454 is a dupe of this bug -- they seem to be in two different providers.

Comment 3

[Joey Minta]

(In reply to comment #2)
> Joey, you might want to take a look at this w.r.t. your other performance
> fixes.
Yeah, the cache I submitted helps a lot here.  There are still some optimizations for the actual recurrence expansion that I've discussed elsewhere, but the cache should reduce the time these calls take by about 80%.  See bug 366177.

Comment 4

[Simon Paquet [:sipaq]]

I've posted some results of a Venkman profiling session in the developer newsgroup. You can find those here:

http://groups.google.com/group/mozilla.dev.apps.calendar/msg/20c4823deb717199
http://groups.google.com/group/mozilla.dev.apps.calendar/msg/a316e5a3539351a3

Comment 5

[Michiel van Leeuwen (email: mvl+moz@)]

The calendar you used in that test doesn't contain any RRULE, so no recurring events. Sine this bug is about recurring items, it's not relevant here.

Comment 6

[martin wessendorf]

This is less a comment than a whine by a user: Sunbird 0.7 ( Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.8pre) Gecko/20071023 Sunbird/0.7 )on my Dell Inspiron 1100 running XP sp2 is abysmally slow.  I have a large calendar with lots of recurring events.  It will take 20 to 45 seconds to perform some simple operations like changing from one week to another.  During this time the program will use 93-98% of the processor.  

Thanks for your efforts, folks!

Comment 7

[Simon Paquet [:sipaq]]

Not going to happen for 0.8.

Comment 8

[Len Weisberg]

My experience on Sunbird 0.7 , on a somewhat old machine
   (Win2K, 512MB, 900Mhz) was exactly as described in Comment #36,
   even though I have only about 60 recurring events 
   (670 events in all).

Simon wrote (Comment #7):
  "Not going to happen for 0.8"

I had assumed that meant that work on this bug was not going to
happen in 0.8 .  But now I wonder if what he meant was that 
this slowness wasn't going to happen.  (Which did you mean, Simon?)


In any case,  *something* sure happened!
  The release notes for Sunbird 0.8 include:
> Sunbird contains several performance fixes,
> which should improve working with larger calendar files

I don't know the details of what was fixed, but the result for me
has been dramatic.  The "simple operations" that were "abysmally slow"
are now, for me, taking 2-4 seconds.  Sunbird's usability has 
improved enormously.

I wonder how this looks for very large calendars, and
I suggest that those concerned with this slowness might want to 
weigh in on whether it is still a problem.
Of course, if there was something in the works to make response even
snappier and more scalable, that's great.  But perhaps the priority
should be re-evaluated.

Thanks for the great work!

Comment 9

[Len Weisberg]

Whoops, I meant Comment #6, not 36

Comment 11

[Paul Smith]

This sounds very similar to my problem...

It becomes very much worse with multiple iCals on a shared network drive.


https://bugzilla.mozilla.org/show_bug.cgi?id=462283

Is it a combination of excessive screen redrawing and inefficient caching and loading of shared remote iCals ?

This is really important to fix, all the good features in TB are sadly worth nothing if our PC's become unusably slow. Shared iCals over local networks that have fast local machine "smart" caches are critical to TB's future success. 

Any ideas/comments from programmers? Does anybody know if this problem is being addressed in Lightning/TB 3.0 integration? Can I help?

Comment 12

[Paul Smith]

I just tried Calendar scope from http://www.calendarscope.com/index.html?src=cs-main&version=4.0r, shared iCals over a network drive...

Super fast, virtually no delay over the LAN.  I may be purchasing Calendar Scope until Lightning/TB 3.0 is fixed to be as fast. This hurts as I love TB and lightning, but this bizarre slowness is absolutely crippling. ( sad )

Good Bye dear friends

Comment 13

[Bas van den Bosch]

How do the fixes for bug 390036 work?

Comment 14

[Philipp Kewisch [:Fallen][📱📆]]

Unfortunately this won't make 1.0 unless someone steps up for it. 

I originally wanted to respond here, but due to the number of comments here I decided to file a new bug for it. Please leave the bug clean for technical discussion, we all agree that performance improvements are needed.

The bug I filed is bug 498968. This bug talks about a few ideas how we can improve performance of the storage provider, especially regarding recurring events.

Comment 15

[Philipp Kewisch [:Fallen][📱📆]]

Ok, so I tried to move the parsing of events into threads, which now actually works when everything is marked as threadsafe.

I couldn't notice an enourmous performance win, the UI is still blocked. Maybe in combination with bug 501302 it works better.

Comment 16

[Philipp Kewisch [:Fallen][📱📆]]

Actually I take that back. I was surprised how error-free things were until I noticed that my C++ files didn't compile and it was using the old version!

"JS Objects cannot be shared across threads"...well so much for using chrome workers at all, this will get worse when we use js-ctypes.

Comment 17

[Wayne Mery (:wsmwk)]

Geoff,

Only comment 0 is relevant in this bug - is this covered in another bug report?

(In reply to Domingo Alvarez from comment #0)

...
Whe populates with several recurrent events like using the
http://ical.mac.com/rainbowfly/FLY-RADIO32SCHEDULE.ics it becomes unusable,
too slow, I did some research and found that in the function getItems from
local storage if we insert a this.mDB.beginTransaction() at the begining and
a this.mDB.commitTransaction() at the end we can reduce the time spent to
half but it is still to slow, when moving fron one day to another it seeks
and proccess 200 recurrent records to show only 6 events in the calendar and
other six in the top list filtered by today events.

It seems that the actual algorithm is not apropriate because it get worst
when the number of recurrencies grow.

For comparison only I downloaded Personal calendar from
www.calendarscope.com the shareware download and importesd the same calendar
from Fly Radio and it's performance is astonishing, if we can solve this
problem the claendar will be usable, because right know it's only a proof of
concept, a good one but not usable.

Reproducible: Always

Steps to Reproduce:
1.import a calendar from
http://ical.mac.com/rainbowfly/FLY-RADIO32SCHEDULE.ics
2.try to use it and navigate through it
3.comment a line on calStorageCalendar.js and see the difference, there is
the slowdown
for each (var evitem in resultItems) {
/////// count += handleResultItem(evitem.item, evitem.flags, Ci.calIEvent);
if (checkCount())
return;
}

Part of this code was added in bug 366177 ?

Comment 18

[Geoff Lankow (:darktrojan)]

Well, obviously, if you completely ignore recurring items and don't display them anywhere the program is going to get faster. That's what commenting the line mentioned does.

However, in my experience there is a lot of unnecessary calculation done, because any recurring event that no longer happens is still being calculated fully. I'm currently considering a number of options on this front.

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to Geoff Lankow (:darktrojan) from comment #18)

Well, obviously, if you completely ignore recurring items and don't display them anywhere the program is going to get faster. That's what commenting the line mentioned does.

However, in my experience there is a lot of unnecessary calculation done, because any recurring event that no longer happens is still being calculated fully. I'm currently considering a number of options on this front.

Is there anything left here after the many fixes of the past 4 years https://mzl.la/3So40M6 (most of them yours)?
Or perhaps covered in another open bug report https://mzl.la/4b7Mjrq.

Comment 22

[Philippp]

Yes we do still have performance problems, hoping for a solution.
Setup:
We are using Thunderbird (usually auto-updating to be on current official release) in a workgroup of ca. 15 people, each person has a calendar shared for all others, and all calendars are configured in each person's Thunderbird. So they all have ca. 15-20 calendars in their Thunderbird visible. (I use a script to roll that out automatically when a new person joins the group.) The caldav DB is on a local Synology NAS with SSD read cache, so I suppose the caldav server is reasonably fast (or at least not unusually slow). I actually have added the SSD cache solely in order to try to alleviate the Thunderbird calendar performance problems, but it did has made a noticeable difference.
On average, there are probably about 1-2 calendar entries per day, in each calendar. We have this scenario for a few years, and the performance issues have gotten worse over time (with growing number of people and growing number of calendar entries (over a few years)).
The calendars are configured with cache mode disabled (this was the default and I had read that lightning cache was still in beta at the time I have set it up, and I did not dare to enable the cache because reliability of the calendar data is essential here).

What we see is very frustrating at daily work:
Listed in descending order of annoyance:

1. Typing text in emails occasionally freezes for about 1-2 seconds!!
2. Scrolling through emails occasionally freezes for 1-2 seconds.
3. When we start TB, it takes about 10 seconds for the calendar view (week view) to fully populate. We can literally watch how it loads the first one or two calendars, then refreshes the whole view, then loads the next one or two, again refreshes the whole view, then the next 1-3 calendars, then again refreshes. So my gut feeling is that the number of calendars and entries scales with a higher order of these numbers.

When we remove all calendars from Thunderbird, these problems are gone.

My impression is that the background process that populates/updates the calendar view is partly blocking the email UI.