436641 - Sends too big query to server if there are many events (over 1000) causing all events to disappear

Status: RESOLVED FIXED

(Calendar :: Provider: CalDAV, defect)

Description

[Keigo IMAI]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_3; en-us) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.1 Safari/525.20
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; ja-JP-mac; rv:1.8.1.13pre) Gecko/20080331 Sunbird/0.8

CalDav provider can also work with iCal server (which is bundled with Leopard server).
However, if you make a big amount of events -- over 1000 events -- then all events disappear from the calendar window.

In the server log (/var/log/caldavd/error.log):
2008-05-14 17:32:06+0900 [-] [caldav-8009] [-] 'Too many matching components in calendar-query report'

The size of query exceeds the specified limit of the iCal server. If I increase the limit (in max_number_of_results = 1000 in /usr/share/caldavd/lib/python/twistedcaldav/method/report_calquery.py), the error does not occur and all events reappear.

This limit is specified in the CalDav specification, accoding to this post:
http://discussions.apple.com/thread.jspa?messageID=7128137#7128137

So please fix the Sunbird to split a big query into a few queries with reasonable size.



Reproducible: Always

Steps to Reproduce:
1. Add iCal server caldav calendar in Sunbird. (http://www.jonathansaggau.com/blog/2007/11/leopard_server_caldav_and_mozi.html)
2. Create over 1000 events in the calendar.
3. All events will disappear.


Actual Results:  
All events disappear from the calendar window.

Expected Results:  
Fix the Sunbird to split a big query into a few queries with reasonable size.

Comment 1

[Roland Felnhofer]

I can confirm this. It is also reproduceable on Windows with an iCal Server running under Linux.

Comment 2

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

I wonder if this also happens on a PROPFIND, in light of bug 455262. I suspect so, therefore confirming this bug. shouldn't be too hard to split the queries into more than one. Setting dependancy since we want to land the PROPFIND code first.

Comment 3

[Ivan Baldo]

This is pretty serious because it is easy to have a calendar with more than 1000 events in a short time if one uses the calendar daily and with various items per day (I got caught by this error and my only workaround is to delete old items which is not very desirable).
Any chance this could be fixed for Lightning 0.10 or we must wait for the 1.0 release?
Thanks.

Comment 4

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Is there some way the client can find out how many events the max is? This might vary depending on server. Also, we need to know for what type of requests we need to watch for this limit.

Comment 5

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

simon_at_orcl noted that most clients split the requests into batches of 100. I think we should do this too, maybe we can make the number of events per request a hidden pref.

Comment 6

[Simon Vaillancourt]

Attachment: multiget should now be done in batches of 100, configurable with the calendar.caldav.multiget.batch.size hidden preference.

Did some basic tests to make sure the batch retrieval works but it will need to be tested on an apple server with +1000 events.

Comment 7

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 436082 [details] [diff] [review]
multiget should now be done in batches of 100, configurable with the calendar.caldav.multiget.batch.size hidden preference.

>+        let batchSize = cal.getPrefSafe("calendar.caldav.multiget.batch.size", 100);
I'd suggest calling it caldendar.caldav.multigetBatchSize, or do you think its likely we will have more multiget-related or multiget.batch-related parameters?


Otherwise r=philipp, I'll leave the final decision on the prefname to you.
Go ahead and check this in any time :-)

Comment 8

[Simon Vaillancourt]

pushed to comm-central with "caldendar.caldav.multigetBatchSize" as the preference name: 
http://hg.mozilla.org/comm-central/rev/e66f19ee877e