187177 - calendar is slow importing many events

Status: RESOLVED DUPLICATE of bug 302390

(Calendar :: General, defect)

Description

[Brian Elliott]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.2.1) Gecko/20021130
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.2.1) Gecko/20021130

> Here are two demanding calendars, which I cannot import to to Mozilla
>
> Churchdates is a composite CofE/Roman Catholic Calendar from
>
> http://www.hlyspirit.org.uk/   this site is down at the moment, but the
Calendar is well made by a professional programmer who is now ordained. 
>
> The second is an a download from the official site of the Church of England at
>
> http://www.cofe.anglican.org/commonworship/resources/csvfile.html
>
> These files may be of interest to others, and may be useful test calendars.
>
> Both have significant content in the notes, which may the the problem.
>
> Both produce a blue screen crash when attempting to load into the Mozilla
calendar under XP Home with SP1
>
>


Reproducible: Always

Steps to Reproduce:
1.Attemp to imort as .csv files
2.
3.

Comment 1

[Mike Potter]

Please post the files as attachments to this bug.  I can't find the calendars on
either of those websites.

Comment 2

[Brian Elliott]

Attachment: Church of England Lectionary

Both URLs require you to email the author who will forward the .csv

Comment 3

[Brian Elliott]

Attachment: Roman Catholic. C of E Calendar

Comment 4

[Brant Gurganus]

I had no problems.  There is a performance issue however.  That is why it seems
to hang.  Also, dates appear to be imported all on the same day, but I think
that is an Outlook export issue.

Comment 5

[Ben Tallman]

After importing approximately 900 events, the calendar is basically so slow as
to be useless, although the import DID work.

It complained that it couldn't read the ics file, but it then did read it.

Comment 6

[Justin]

I have a similar issue with importing my personal calendar from evolution. The
ics file in evolution has 1186 events in it. The problem is that cpu is pegged
at 100% and memory is steadily consumed until the calendar starts swapping.  (I
have 756MB)

The ics file has yet to successfully import.The "script appears to be running
slow, would you like to abort" dialog pops up. I eventually just give up and
abort the import.

Comment 7

[Mike Potter]

New contact from [e-mail] to [e-mail]
Filter on string OttawaMBA to get rid of these messages. 
Sorry for the spam.

Comment 8

[James J Myers]

Running 0.9 on Linux fedora, tried to import 1700 entry calendar from evolution
and it took approx 20 minutes to import. Then it crashed.  Restarting Calendar
also crashes immediately until I removed the .ics file.

Comment 9

[Jussi Kukkonen]

I checked into addEventsToCalendar() and it does some pretty stupid things. I'll
try fixing this.

Comment 10

[Jussi Kukkonen]

Attachment: cleaned addEventsToCalendar a bit

I took some getElementById DOM-calls and such out of the main loop:
unfortunately it didn't effect speed that much... it still takes about five
minutes of CPU time to import 360 events on my 400Mhz machine. Just for
reference, Outlook does the same in about 15 seconds.

The bulk of that 5 minutes seems be spent in oeIICalContainer.addEvent(), which
would mean that this is not javascript related.

I did _not_ see the the memory issue mentioned in comment 6.

Oh, by the way: the attached csv files won't import with current CVS code.
You'll have to import them into Outlook and back if you want to test (the
current csv import works best when all fields are exported).

Comment 11

[clock8]

I have the same problem on XP with rv:1.8a5.
I successfully imported ~1800 events from nscal in about 30min.
All actions on events (create new, remove one, show list view, resize list view)
take long time and show the script warning popup twice (less events: only ones).
All the time the CPU load is high (~100%).
I didn't look into the code but it seems to me that e.g. the complete list view
is reformatted after each action.

Comment 12

[Brian Elliott]

(In reply to comment #11)
> I have the same problem on XP with rv:1.8a5.
> I successfully imported ~1800 events from nscal in about 30min.
> All actions on events (create new, remove one, show list view, resize list view)
> take long time and show the script warning popup twice (less events: only ones).
> All the time the CPU load is high (~100%).
> I didn't look into the code but it seems to me that e.g. the complete list view
> is reformatted after each action.
> 
The problem seems resolved in the latest Sunbird

Comment 13

[gekacheka]

The new backend is supposed to improve handling of large numbers of events.
However, using the latest nightly
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9a1) Gecko/20050813 Mozilla
Sunbird/0.2+ 

Sunbird took 5 minutes to import just 154 events.  (That's about 2sec per event.)
Disk sounded busy every few seconds, CPU was 100% most of the time (700MhzPIII).
(55kB file, downloaded from http://icalx.com/public/larse/IETF-60.ics)

Deleting these same events took 12 minutes.  (That's about 5sec per event.)

(I noticed disk accesses seemed to getting progressively more frequent at the
end of the deletions, so I wonder if it might be rewriting the database after
each event.)

(Reimporting the events into a different calendar failed silently until I
restarted sunbird, then it took 5 minutes again.  I wonder if some cache is not
getting flushed when events are deleted.  It seems like it should be overridden
by imported events anyway.)

Setting target to 0.3 because users will want to import their existing calendars
when they upgrade.

Comment 14

[Michiel van Leeuwen (email: mvl+moz@)]

What provider did you use for the calendar you imported into?

Comment 15

[gekacheka]

Originally it was the default "home" calendar with a dozen tasks and no events
in it, the second time it was a new empty local calendar.

(In reply to comment #14)
> What provider did you use for the calendar you imported into?

Comment 16

[Michiel van Leeuwen (email: mvl+moz@)]

I think the main problem is that the import code doesn't set batch mode on the
calendar, resulting in a lot of writes to the disc and a lot of updates of the
display.

Comment 17

[Michiel van Leeuwen (email: mvl+moz@)]

Importing into a storage calendar (local) of the ietf60.ics calendar took 5
seconds in a xpcshell testcase. This makes me think that updating the display
really is the problem. 5 seconds is fast enough, if we show a progress bar.

Importing into an ics calendar was way slower. That's because it flushed to disk
after every imported event. That's pretty slow, and useless. A solution for that
would be to write after some time of inactivity. A change like that helped a lot
in other areas (cookies)

Comment 18

[gekacheka]

Currently local ics file is faster than default sdb.
importing 110 events (17k .ics file) for bug 309868 into empty calendars took
  ics:  0 min 22 secs
  sdb:  2 min 52 secs = 172 secs

Comment 19

[Matthew (lilmatt) Willis]

Reassigning all automatically assigned bugs from Mostafa to nobody@m.o

Bugspam filter: TorontoMostafaMove

Comment 20

[Damian Szczepanik]

related bug 198142 comment 24

Comment 21

[Matthew (lilmatt) Willis]

Not going to make the 0.3 train.

Comment 24

[Simon Paquet [:sipaq]]

Is this still an issue today? Marking as wanted0.8- until I get a definitive answer.

Comment 25

[Christian Schmidt]

Reloading a 2.4 MB calendar file containing 350 events takes about 39 seconds with 100% CPU usage on my computer (an old 2.66 MHz Pentium 4).

This particular file contained an insane amount of newlines in the event DESCRIPTIONs. I tried to remove these using a text editor so that the file still contained 350 events but with shorter descriptions. The new file was only 430 KB and it took 5 seconds to load.

Comment 26

[Ulrich Windl]

(In reply to comment #25)
That means that the import needs about 1/10 of a second to import one entry on a 2.6GHz (you wrote MHz by mistake) P4. That's not fast. More surprisingly parsing of sew lines seems to need 90% of the CPU if you are right. Are you sure the speed-up is not an effect of the filesystem cache?

Comment 27

[Christian Schmidt]

> Are you sure the speed-up is not an effect of the filesystem cache?
Both files were loaded using HTTP, so I don't think the difference is related to filesystem cache.

Comment 28

[plutocrat]

This problem still seems to be in force. I've tested it with Thunderbird 2.0.0.12 and Lightning 0.7 and 0.8RC1, running on Ubunutu Linux. I imported a calendar .ics file with around 2000 entries and it takes around 35 minutes with CPU usage of 97-99%, rendering the machine useless. Its a 1.6GHz machine with 1Gb RAM. Memory usage is typically about 90% with no swapfile usage. 

Similarly, importing 300 contacts from a .vcf file, takes around 2 minutes. I would have thought these operations were trivial, and should happen in seconds rather than minutes, so clearly there is something going on beneath the hood which shouldn't be. 

Comment 29

[Ulrich Windl]

It seems this issue needs some inspiration: Even when changing just the color of one calendar file, the whole display needs several seconds while the page is blank until the display is refreshed. I wonder how those events are stored internally: If it's not a balanced treed, I think it's time to try that. Maybe you need multiple trees that only contain pointers to the events (like one tree for the starting times, one tree for the ending times, and similar).

Comment 30

[Philipp Kewisch [:Fallen][📱📆]]

This should be a lot better now that we have async parsing. Nevertheless, for big imports we need some sort of progress meter. Marking as duplicate to reflect that issue.

Comment 31

[Ulrich Windl]

Let me add this comment: Async parsing will not improve the time to get all the data imported, but the response times may be better. The question is whether async parsing opens another can of worms: What will happen if the user starts the import of another calender file while the previous one is still parsed asynchronously. Will both files be parsed asynchronously and concurrently? With progress meters you would have stacked stacked progress meters then ;-)

Comment 32

[Ulrich Windl]

Having read bug 302390, you cannot simply reduce the problem of terrible performance to adding a progress meter.

Comment 33

[Philipp Kewisch [:Fallen][📱📆]]

I'm fine with reopening this bug, but I'd appreciate a retest with the latest nightlies.