Open Bug 1543953 Opened 6 years ago Updated 7 months ago

REPORT request receiving time too high (0.5 to 3 seconds per request). Loading one ~4000 events network calendar, both online and offline, more than 1 minute and 46 network requests. Adjust calendar.caldav.multigetBatchSize preference?

Categories

(Calendar :: Provider: CalDAV, defect)

Lightning 6.9
Unspecified
All
defect

Tracking

(thunderbird_esr128 affected)

Tracking Status
thunderbird_esr128 --- affected

People

(Reporter: richard.leger, Unassigned)

References

(Blocks 2 open bugs)

Details

(Keywords: perf)

Attachments

(14 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

Start Thunderbird 67.0b1 (32 bits) for Windows with patch.v03 applied from Bug 1502923 with only one network CalDav calendar setup containing ~4000 items. No email setup. New profile. Before closing TB and restarting TB to measure, Find Events view was set to show All Events and sort items by Title column header.

[This bug was observed/uncovered while troubleshooting Bug 1502923]

Actual results:

Via TB devtools > Network - ~46 requests observed to load the all calendar - mostly REPORT request. The first few REPORT requests take the longest in term of wait and receiving response time - can varie from 0.5 seconds to 3 seconds in most cases (but has been observed sometime up to 12 seconds response time for one request!!!)

Via Task Manager > Performance > Wifi - You can see the numerous requests spreading over more than 1 minute time to complete the load of just one network calendar in TB.

Expected results:

The initial loading of the calendar data shall be much faster than observed and which much less network requests in number but maybe bigger in size.

When loading the entire calender via GET request as an .ics file it only takes a fraction of a second... so definitely not a calendar server issue here...

Find attached the Task Manager > Performance > Wifi graph showing the minute of request this was capture just before the calendar end loading... the all graph corresponds to most of the network requests used for loading the calendar...

Attached image devtools.network.png —

Find attached the devtool > Network result - Too many requests - Too much time for each!!!

I suspect the request send by Lightning are not performant as they seems to look for each items and load each items in turn while it should just get the calendar as .ics file in one request and parse it (all items) to initialise the Calendar... that would be a much faster way to do... but may not be possible due to the nature of DavCal and the way it works.

FYI, find attached the calendar setting options (all unticked) used to run the tests.

Also increasing the calendar.caldav.multiget.batch.size option from 100 (default) to 200,400,600,1000 make no positive difference. Reducing it either...

Component: Untriaged → Provider: CalDAV
Product: Thunderbird → Calendar
Version: 67 → Lightning 6.9

Currently at startup, Thunderbird run a CalDAV sync on the calendar which mean:

  1. Get calendar feature and options
    ==> few first requests (x3)
    in less than one second

  2. Get a list of all etags and uri of EACH calendar objects (VEVENT, VTODO, etc...) - meaning ALL of them
    ==> 1 REPORT request - Which return an XML file of 32000 lines for ~4000 items!!!
    in ~12 seconds

  3. Send requests to sync each items one by one (really inefficient in the case where ALL calendar items need loading) by chunck of 100 (default value)
    => 42 REPORTS requests!!!
    in ~1 to ~2 seconds each

I think it would be more efficient to do the following at TB startup:

  1. Get calendar feature and options
  2. Get the full calendar as .ics file via a GET request instead and parse it
    This take only 2seconds with my 4000 items calendar
  3. Get the etag and uri for each calendar objects

and/or use a time range query to sync only the past 60 days and future 365 days at boot up... leaving the rest to be synched later on either when TB is idle or when user request it via the UI interface (time frame).

  1. Get calendar feature and options
  2. Get a list of all etags and uri of calendar objects (VEVENT, VTODO, etc...) within -60 days +365 days <-- this would be the initial timespan
  3. Send requests to sync each items within the time range one by one by chunk
    3bis - Reduce the quantity of data to be transferred by selecting only the minimum properties needed by the UI elements (e.g Title, Start Date, etc...) and disregarding what is not required (e.g Description, Attendees, Attachments, etc... - Those only need to be loaded when end-user edit the VEVENT or VTODO... which is already the case...)

Later on when idle (or upon request by end-user)

  1. do the same on the time frame -beginingOftime -60 days
  2. do the same on the timr frame +365 days +endOfTime(infinite)

As feature request, I would also suggest to provide additional UI calendar properties to end-users to define the following on per calendar bases:

  • chunk size - people with high speed connection could benefit from having higher chunk size to reduce the number of requests...
  • initial sync timespan past (e.g 60 days)
  • initial sync timespan future (e.g 365 days)
    I have created Bug 1544375 "Ligtning - Add network calendar UI options: Chunk size, Initial sync time range filters (sync timespan future and past)" to deal with it separately.
See Also: → 1502923

(In reply to Richard Leger from comment #7)

3bis - Reduce the quantity of data to be transferred by selecting only the minimum properties needed by the UI elements (e.g Title, Start Date, etc...) and disregarding what is not required (e.g Description, Attendees, Attachments, etc... - Those only need to be loaded when end-user edit the VEVENT or VTODO... which is already the case...)

Here is an example of minimum properties that may be required, but I don't know if it is possible within CalDAV protocol to limit data to only those properties at least during initial sync at startup... it seems to be possible according to RFC but haven't clearly understood how that would work as calling the calendar-data fields means you get all the data from the object including properties that you may not need (e.g description, attendees, attachments)... maybe possible via the same REPORT query that allow to get the etag and uri... to be explored perhaps... that would increase the query response size but may be workable if associated with a filter time range...

TB VEVENT

Calendar Name
UID
DTSTAMP
???ORGANIZER
Title SUMMARY
Start DTSTART
???Duration DURATION
Category CATEGORIES
Location LOCATION
Status STATUS
Reminder VALARM
Priority PRIORITY

End DTEND

TB VTODO

Calendar Name
UID
DTSTAMP
???SEQUENCE
Title SUMMARY
Start DTSTART
???Duration DURATION
Category CATEGORIES
Location LOCATION
Status STATUS
Reminder VALARM
Priority PRIORITY

Done
Due DUE
Due in
Completed COMPLETED
%Completed PERCENT-COMPLETE

Also TB developers shall keep in mind that the first thing TB users do after starting Thunderbird is to receive and read emails... email processing/actions shall prevail on calendar processing... and not the opposite way around as currently it is the case...

Currently at startup, even if I receive emails in my IMAP mailbox (no cache), I cannot read new email till all the calendars have been initially loaded (no cache either)... If I click on new email, it does not load in preview pane (or in new tab) till the calendars are fully loaded first (which takes 5mn)... that should not happen in ideal order of things :-)

Loading an IMAP email shall take priority/precedence over calendar loading/sync... as TB is more "muli-processing" now, that should become more possible...

This is also not helped by heavy mail folder activity at startup, causing lots of IO.

Much of said IO being on main thread, i.e. UI thread, and being synchronous IO - Bug 588952 - autosync should use new async folder opening to avoid extremely bad performance at startup or when reading messages with very big IMAP folders (like gmail All Mail)

Keywords: perf
See Also: → 1544375, 1247240
Summary: Lightning network REPORT request receiving time too high (0.5 to 3 seconds per request) - Loading one ~4000 events network calendar takes more than 1mn - 46 network requests required (isn't it too much?) → Lightning network REPORT request receiving time too high (0.5 to 3 seconds per request) - Loading one ~4000 events network calendar takes more than 1 minute - 46 network requests required (isn't it too much?)

(In reply to Wayne Mery (:wsmwk) from comment #10)

This is also not helped by heavy mail folder activity at startup, causing lots of IO.

The problem is that calendar loading "hijack" the main thread/UI so nothing else happens till calendar(s) loading has completed. You cannot access content of new emails for example. If you click on email, the preview pane remain "empty" the email content is not loaded... Not convenient, not "normal" behaviour expected... :-)

The Bug 1502923 also does not help in this regards as it drastically delay calendar loading, it could be added as blocking Bug 487832 "(tb-startupperf) [meta] Thunderbird startup performance issues"..

Much of said IO being on main thread, i.e. UI thread, and being synchronous IO - Bug 588952 - autosync should use new async folder opening to avoid extremely bad performance at startup or when reading messages with very big IMAP folders (like gmail All Mail)

It there a way for Lightning to also load CalDav calendars asynchronously? Using workers perhaps or else async feature of TB?
This way the calendar could load slowly each in a different thread ideally (or one dedicated calendar thread) in the background while keeping email feature available (e.g end-user reading new email received) at all time especially at startup where email may be the primary feature end-users access to read new emails...

Will the integration of calendar into Thundebird help reduce any of this responsiveness? In other words, since it will all be one code base, will that make it any easier to separate out the I/O or do the prioritization you are trying to do? If nothing else, this bug bug 1493008 should be noted here.

(In reply to Worcester12345 from comment #12)

Will the integration of calendar into Thundebird help reduce any of this responsiveness?

No - integration changes nothing and will not help. Not even the analysis.

(if you're going to post bug numbers please post as "bug NNN", not a link)

FYI, still 46 network requests in TB.70.0b2 to load one calDAV network calendar with about 4000 items...

Attached is devtools network request part 1...

... part 2 attached...

See also Bug 1572823...

May be time to look into this bug because reducing the number of network requests may drastically help reduce the time it take to load network calendar... (new test results available here https://bugzilla.mozilla.org/show_bug.cgi?id=1572823#c16 and here https://bugzilla.mozilla.org/show_bug.cgi?id=1572823#c19) each network request shall be able to load more than 100 items at a time especially on fast networks e.g 1GB connection.

The best may be for TB

  • to detect speed of network connection and adjust number of items it loads per network request consequently
  • or to let end-user choose the network request batch size...
  • or both!
See Also: → 1642292
Attached image image.png —

FYI and the record...

As observed in TB 78.0b1 (64-bits), it still takes about 50 network requests to load one CalDAV calendar with 4000+ items...

See attached...

(In reply to Richard Leger from comment #7)

I think it would be more efficient to do the following at TB startup:

  1. Get the full calendar as .ics file via a GET request instead and parse it
    This take only 2seconds with my 4000 items calendar

Am I dreaming or I have been heard? What a surprise today! Yes, Yes, Yes... finally!

Just upgraded to TB 81.0b4 (64-bit) and testing loading my ~4000 items caldav network calendar... while measuring and monitoring I could see only one network request and I first thought... damn calendar loading is broken again?!

...but checking via DevTools > Networking...
...what a surprise!!!

The 50 multi-get request that have been hindering calendar loading performance for so long have now been replaced by one single GET request that load the entire .ics calendar at once!!! Oh my god! What a speed difference max 22 seconds top... (result obtained with new profile, no email, one caldav network calendar newly setup with ~4000 items)

  • Was it intended or an unintended side effect of something else implemented, corrected or removed?
  • Is that gonna be the new way?
  • Would that perf improvement "stick" from now on?
    Can someone confirm?

No info in the release note:
https://www.thunderbird.net/en-US/thunderbird/81.0beta/releasenotes/?uri=/thunderbird/releasenotes/&locale=en-US&version=81.0&channel=beta&os=WINNT&buildid=20200914232954

Nor could I find any changelog in the source code that may explain such change:
https://hg.mozilla.org/releases/comm-beta/log/a8ea452b9f4cc059a4afebaf5d54af4e72e4b5c4
If someone could point out perhaps the changeset related to this change?

If that is intended that is great, great, great news to all TB calendar users out there!
Thank you...

I keep dreaming for now... hoping someone would confirm this reality!

I have yet to test difference on startup performance with my current default profile!

(In reply to Richard Leger from comment #19)

I have yet to test difference on startup performance with my current default profile!

I just re-tested and bad news, my existing already set calendar in my default profile (that I also use for emails) does not behave in the same manner. When I enabled it and run the sync, instead of one GET request as described in my previous comment, it still generate multi-GET requests to sync and therefore still affected by Bug 1642292.

This suggest that to get the new behaviour and related performance the calendar needs either to be deleted/re-created from scratch or that settings in my current already set calendars may need some adjustments to benefit from new behaviour... I would expect TB to take care of that upon upgrade :-)

Any clue?

Flags: needinfo?(paul)

I new it was too good to be true!

Indeed it appears than when testing and obtaining result as per Comment 19, when creating a new profile and adding my network calendar, it was added as an ICS type calendar (read-only) not allowing me to choose CALDAV (read-write capable) while my pre-existing calendar in my default profile was in the past setup as a CALDAV calendar... that explain the difference in loading performance... and the one GET request (20s) for ICS vs multi-get REPORT requests (7+mn) for CALDAV...

Bottom line... back to square one!

I guess the fact that I cannot choose CALDAV type of network calendar when adding a new one in TB 81.0b4 (64-bit) on Windows is a new bug in itself :-)

Hm, this could be somewhat related to the work in bug 1663399. That is troubling that you expected a CalDAV calendar but got an ICS one instead. Can you please open a separate bug for that, with the steps to reproduce? Like, if possible can you indicate what you entered for username and location when setting up the calendar? Or if not the exact text, then the general shape of it? That may be causing it to detect ICS rather than CalDAV.

Flags: needinfo?(paul)

(In reply to Paul Morris [:pmorris] from comment #22)

Hm, this could be somewhat related to the work in bug 1663399. That is troubling that you expected a CalDAV calendar but got an ICS one instead. Can you please open a separate bug for that, with the steps to reproduce?

New Bug 1666036 has been created as per your request.

See Also: → 1666036

Needs new version:

" Version:
Lightning 6.9"

Still affect 78.x ESR and beyond...

(In reply to Richard Leger from comment #25)

Still affect 78.x ESR and beyond...

Have you tested with 82.0b2? 82.0b3 has been built but not yet made official on tb.net so maybe hold off a few days and test with that when it's released.

Blocks: 1649103

Richard,

How is this behaving in 91.0b6?

(In reply to Arthur K. [He/Him] from comment #27)

How is this behaving in 91.0b6?

Will test again and revert back... as preliminary result I still see 52 to 82 network requests... but some of them may be telemetry or else... need to isolate the caldav part...

Each .ics event seems to still be fetch by group of 100 per network request, while they could be loaded in a much higher number to reduce the number of network request and accelerate the loading process...

Definitely this number shall be adjustable either per calendar property or automatically adjusted automatically based on network speed detected... my previous attempt to adjust this number manually in TB code failed so I haven't got a clue where that value is currently setup in TB to test with higher number... to reduce the number of network request required... one network request can definitely handled more than 100 .ics items data at a time when loading response from the server...

Also I question the necessity to download the entire .ics data including description and other part that may not necessarily be required to load until the item is edited or viewed in the calendar especially in online mode... as it increase the size of the response necessarily... to just show items in the calendar view, you only need to load the current view period of item... on demand that would be much quicker and efficient... I believe...

Flags: needinfo?(richard.leger)

(In reply to Richard Leger from comment #28)

(In reply to Arthur K. [He/Him] from comment #27)

How is this behaving in 91.0b6?
Will test again and revert back...

I have tested again in TB 91.0b6 (64-bit). The situation remain the same.

58 Requests for 10MB+ data in 1.90mn according to devtool networking for loading one CalDAV calendar containing 4000+ items.

Stopwatch shows calendar takes about 2mn to load (calendar items appear in the view only after that time lapse) with default TB settings (unifinder set to Event in the next 7 days - no column selected for ordering). No pause observed between network requests.

Setting up unifinder to All Events and ordering by Title column still increase drastically the time it takes to load the calendar because after each batch TB order the least instead of waiting for all batch to load prior ordering the list. The number or network requests remain the same but some pause appears between them at some point... as TB is processing previous request...

I believe that increasing the batch size from 100 to 500 or 1000 would reduce the number of network requests and accelerate the loading of the calendar. But I don't know how that would impact TB to parse each .ics items in such high quantity. In theory it shouldn't but I haven't been able to try.

I hope that make sense.

Flags: needinfo?(richard.leger)

What percentage of these 4000 events are recurring items?

Flags: needinfo?(richard.leger)

Out of 5918 UID ical items, about 205 with RECURRENCE and 89 with RRULE (27 DAILY, 16 WEEKLY, 28 MONTHLY).

Flags: needinfo?(richard.leger)

2566 with DESCRIPTION (often converted from email as event or task) among with 50 with X-ALT-DESC;FMTTYPE=text/html.

Richard,

What version of TB are you running now?

(In reply to Arthur K. [He/Him] from comment #33)

Richard,

What version of TB are you running now?

I was on 98b3 and now have moved to 99b1 but having issue with the calendar especially items not displaying in the week view and when adding items... but that is a separate issue from this bug :-) I may have notice also possibly an issue with reminders, with default selected calendar upon restart of TB, etc... but I need to test more about all those... also I'll wait for later beta as improvements may come already...

Apart of this hiccup, I noticed the calendar load much faster (well done!) than before but nothing is displayed in the week view upon first loading of calendar for about 1mn37s (Did a quick first time testing in beta 99b1 with no email, one calendar, all events, sorted by title - to be re-confirmed) the time it takes for the calendar to load completely in TB. I can see some items appearing progressively in today pane but they disappear all for while just before the calendar items appears in the calendar view... (that was the case already before if I recall correctly)

Currently I can see .ics items are still loaded by batch of 100 so it requires about 62 network requests sill to load the entire network calendar...
Can't the batch limit be increased to 500 or be automatically calculated based on the network speed so on low speed network if could be low but in high speed network be increased? I really don't see any reason for the batch of items to be limited to 100 items per network request!
It might even work with higher batch of 1000 or more... reducing the number or network request would increase the speed of loading...

My entire .ics caldav calendar is taking less than 2 seconds max to load in one go via https! So I am sure that TB can do better than 1mn37 sec to load calendar... it did in the past... ~1mn at the best...

All this in online mode, no Offline Support.

Richard, how is the current beta for you?

Flags: needinfo?(richard.leger)

Hi Wayne,

There have been slight improvement in the sense that there is no longer stopped gap between network requests required to load a CalDAV calendar... all the requests now run one after another... but as you would see below, much progress still needed...

In TB 105.0b3 x63 on Windows it takes about 1mn50 to load one CalDAV calendar with about 5000 items with unifinder set to show events in the next 7 days... or all events without any column ordering... So the situation has deteriorated a bit... as before it was taking about 1mn for 4000ish items...

If I select All Events and order by Title column in unifinder the delay increase to about 2mn... to load the same calendar as for each network request TB order the list again of loaded items... it was already mentioned before that when loading the calendar first time, the list do not need to be ordered for each network request it could be done ones the all calendar is load (order only once instead at each network request) which is much faster.

Broadband speed: 4.49Mb Download / 1.02Mb Upload

The number or network requests required has increased to 67 which is expected as only 100 items loaded per network request so about 10 more requests than before... which correspond to the jump from 4000 to 5000 items in the calendar...

I still believe that reducing the number of network requests by increasing the number of item downloaded per request (or at one time) would reduce the number of requests required to load the entire calendar and therefore accelerate the process of loading it especially at initial startup of TB...
If 1000 items were loaded for each network request, it would take only four or five network request to load the entire calendar...

Currently it means I need to wait 1mn50 to see all my calendar items... a bit long :-)
Event longer if I have multiple calendars that load in parallel (e.g colleagues or else).

Thankfulness this process no longer prevent to read email... which for most people would be the first task when they startup TB but still if I start TB to join a meeting I still need to wait depending when it will appear in the Calendar view...

Hopefully some of the performance aspect would be tackled in the next stable iteration of TB...

Regards,
Richard

Flags: needinfo?(richard.leger)
Attached image match-settings.png —

Richard,

Don't know if it'll make much difference but I noticed a couple Calendar related changes happened in 105.0b4. Might you try with that version?

Also, please check something for me. I noticed recently that on the properties of a couple of my calendars, there was a mis-match between the Username: and Email: setting. On a couple of my accounts, Email: was set to my primary account instead of the one shown in the Username: field. It seems better now that I fixed it so it might help you out too. just trying to tick all the boxes. =)

(In reply to Arthur K. [He/Him] from comment #37)

Don't know if it'll make much difference but I noticed a couple Calendar related changes happened in 105.0b4. Might you try with that version?

Same in TB 105.0b4 (64-bit) on Windows about ~1m48 to load my entire calendar when first enabled... over a 4G connection...
The problem remain that it requires 62 network request of 100 items each about... to load my entire calendar... Surely TB shall be able to load the entire calendar in one go at startup... within one request... or much fewer!!!
As previously stated, to load my entire .ics calendar it takes only 1-2s max via HTTP...
I believe this is where the next step of performance improvement shall occur...

Also, please check something for me. I noticed recently that on the properties of a couple of my calendars, there was a mis-match between the Username: and Email: setting. On a couple of my accounts, Email: was set to my primary account instead of the one shown in the Username: field. It seems better now that I fixed it so it might help you out too. just trying to tick all the boxes. =)

Thanks for the suggestion, but in my case my username is different from my email address by design, and it is set correctly...

(In reply to Richard Leger from comment #38)

(In reply to Arthur K. [He/Him] from comment #37)

Don't know if it'll make much difference but I noticed a couple Calendar related changes happened in 105.0b4. Might you try with that version?

Same in TB 105.0b4 (64-bit) on Windows about ~1m48 to load my entire calendar when first enabled... over a 4G connection...
The problem remain that it requires 62 network request of 100 items each about... to load my entire calendar... Surely TB shall be able to load the entire calendar in one go at startup... within one request... or much fewer!!!
As previously stated, to load my entire .ics calendar it takes only 1-2s max via HTTP...
I believe this is where the next step of performance improvement shall occur...

This does not exclude possible performance improvement within the Calendar UI/UX widget that may also take time to populate items into the TB interface... which may be another culprit... but this is/or will be work on within the Calendar UI/UX revamped planned/coming up from my understanding... at least I do hope so :-)

(In reply to Richard Leger from comment #39)

Same in TB 105.0b4 (64-bit) on Windows about ~1m48 to load my entire calendar when first enabled... over a 4G connection...
The problem remain that it requires 62 network request of 100 items each about...

This test shows that it takes TB ~1s80 per each network request handled!
Way to long from a computer speed capability... there is obviously a problem somewhere... that need fixing... and reducing the number of request may be one of the pathway to improved TB startup performance for CalDAV...

(In reply to Richard Leger from comment #40)

(In reply to Richard Leger from comment #39)

Same in TB 105.0b4 (64-bit) on Windows about ~1m48 to load my entire calendar when first enabled... over a 4G connection...
The problem remain that it requires 62 network request of 100 items each about...

This test shows that it takes TB ~1s80 per each network request handled!
Way to long from a computer speed capability... there is obviously a problem somewhere... that need fixing... and reducing the number of request may be one of the pathway to improved TB startup performance for CalDAV...

I just installed 106.0b1 the other day and while looking at the relnotes, I saw something which looked promising: bug 1783585

Not directly addressing your issue of network request handling but the messy knot may have gotten one less tangle in it now.

@Arthur,

(In reply to Arthur K. [He/Him] from comment #41)

I just installed 106.0b1 the other day and while looking at the relnotes, I saw something which looked promising: https://bugzilla.mozilla.org/show_bug.cgi?id=1783585

I have seen improvement you refer to as well but that is not gonna fix this bug and delay to load the calendar itself.

...yes, they have been a lot of effort made to accelerate startup perf of TB that I have been watching and enjoying...
We can now read email within 3-4s of opening TB thanks to those efforts, which is great...

...but at the moment I still need to wait a long time for the full calendar to load and to access all my items... and you can imagine that issue get worth if you get multiple calendars setup... which is far from being unusual... so efforts would continue hopefully...

@Geoff,

I just wanted to put this again on your radar, as part of the dev teams on-going efforts for improving TB performance, especially at startup, and especially with CalDAV feature...

Cheers,

Flags: needinfo?(geoff)

The number of events in a single request is set by the pref calendar.caldav.multigetBatchSize and the default is 100. You could try increasing that, but your server may not like it.

Flags: needinfo?(geoff)
Attached image image.png —

(In reply to Geoff Lankow (:darktrojan) from comment #43)

The number of events in a single request is set by the pref calendar.caldav.multigetBatchSize and the default is 100. You could try increasing that, but your server may not like it.

I've just checked in TB 105.0b4 (64-bit) and this option does not exist...

Are you suggesting it should be added manually as a number type?

I tried setting it up to 120 and it seems to increase number of items per request but the over all process is so slow that after 4mn I only reach 18 requests processed :-(

Looking are REPORT request Timing, in Dev Tools > Network, It seems to take 8s to 45s per requests (Receiving:)...

It seems I have 5% packet loss when accessing my calendar server today so probably not the best time to test this... I will fix and test again later on... as test in Comment 44 may obviously be affected this unexpected and exceptional situation!!! :-(
Keep you posted...

Flags: needinfo?(richard.leger)

The REPORT request response seems formatted as below...

<?xml version="1.0" encoding="UTF-8"?>
<C:calendar-multiget xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:prop><D:getetag/><C:calendar-data/></D:prop>
/davical/caldav.php/richard/home/443544bf-649d-4fc1-b757-561cbebd5f3c.ics</D:href>
/davical/caldav.php/richard/home/891b2a90-192f-4d03-b345-1624abe8366a.ics</D:href>
/davical/caldav.php/richard/home/a981b2d8-00a1-44b3-aae7-f73749076154.ics</D:href>
(...)
up to 100 line (or 120) as per calendar.caldav.multigetBatchSize size.

So I don't really see or understand how the server would not be able to handle to provide more lines in the response as you stated "but your server may not like it"...
Server side it is just a postgresql request statement... almost... so it would certainly be able to provide more data in one request... the question may be more how TB would be able to handle more data in a one response...

Also when loading the initial data all item data are downloaded including hidden fields like description/attachments, etc... those could be with delay or in separate async requests at a later stage as at startup you only need to get the data required to fill in the calendar view UI and poping reminders, so titlte,startdate,enddate, reminder,etc... would suffice to load at startup...

For people working in online mode, the description/attachment and other fied show only when item is opened could be requested when item is effectively opened... or loaded at a later stage than startup via a different async process for those that use caching...

Could that help improve the performance of loading CalDAV calendar? Especially at startup? Of course time it takes to populate the item in UI is also to be considered and optimised... but that is already on your radar and is a separate issue...

Flags: needinfo?(richard.leger)

(In reply to Geoff Lankow (:darktrojan) from comment #43)

The number of events in a single request is set by the pref calendar.caldav.multigetBatchSize and the default is 100. You could try increasing that, but your server may not like it.

I know this will sound counterintuitive but I am wondering what happens if he DECREASES calendar.caldav.multigetBatchSize to something like 50.

Richard,

I was curious if you'd ever tried this experiment: using portable TB (https://portableapps.com/apps/internet/thunderbird_portable), set up the account which has the huge calendar on some laptop and go to a public WiFi or friend's network and see if the same issue happens? Just thinking out loud.

(In reply to Arthur K. (he/him) from comment #47)

(In reply to Geoff Lankow (:darktrojan) from comment #43)

The number of events in a single request is set by the pref calendar.caldav.multigetBatchSize and the default is 100. You could try increasing that, but your server may not like it.

I know this will sound counterintuitive but I am wondering what happens if he DECREASES calendar.caldav.multigetBatchSize to something like 50.

Yes no problem I will try that as well... once the ISP issue server side issue is fixed!

Flags: needinfo?(richard.leger)

(In reply to Arthur K. (he/him) from comment #48)

I was curious if you'd ever tried this experiment: using portable TB (https://portableapps.com/apps/internet/thunderbird_portable), set up the account which has the huge calendar on some laptop and go to a public WiFi or friend's network and see if the same issue happens? Just thinking out loud.

Yes I did try on other computers in the past and it did get the same results... I used to do test on ADSL connection but I now use 4G connections which may be faster but may have more latency to see how TB reacts...

The key to see it is to disable the calendar caching... this is the way to get real performance of CalDAV loading especially at startup... clearly things can be improved... something remain wrong somewhere and I am trying to look into it to raise it to dev team(s). This bug being on of those issue.

Changing the default value of calendar.caldav.multigetBatchSize may potentially positively resolved this bug, but so far I doubt it works as expected... and I will soon re-test and report back about it.

Hi Arthur, Geoff,

(In reply to Richard Leger from comment #49)

(In reply to Arthur K. (he/him) from comment #47)

(In reply to Geoff Lankow (:darktrojan) from comment #43)

The number of events in a single request is set by the pref calendar.caldav.multigetBatchSize and the default is 100. You could try increasing that, but your server may not like it.

I know this will sound counterintuitive but I am wondering what happens if he DECREASES calendar.caldav.multigetBatchSize to something like 50.

So I tried to modify the multigetBatchSize to 120,300,1000,20... but in all case the calendar is slow to load...

  • if you increase the number, the number of request reduce but the time it takes to process each request increases...
  • if you reduce the number, the number of request increase, but the time to take to process them reduce but because they are so numerous it still takes time to process them... 500-700ms to process each request seems rather long! Mostly due to Receiving part...

I noticed in the attachment that at multigetBatchSize 20, after the PROFIND request, with the first REPORT request that follows, TB takes 35s to read the entire response from the server (or cache)... what is identified as Receiving time in the request Timing. This may suggest that TB is struggling to parse the XML from the server response somehow... In dev tool the XML response is truncated as too big to display on screen... but maybe the issue is that XML parser form TB... or the time/performance required to process data when received... something looks wrong here...

So until that is fixed, I don't think that changing multigetBatchSize would be a solution for now. But it may have helped highlight the possible problem... maybe something in the response is making the XML parser to chock somehow... or TB processing is not fast enough... but that is clearly not normal I believe... each other request still seems to take a long time to process as well despite size of response reduce to a 20 items per response.

First REPORT Request highlighted in attachment:

<?xml version="1.0" encoding="UTF-8"?>
<sync-collection xmlns="DAV:"><sync-token/><sync-level>1</sync-level><prop><getcontenttype/><getetag/></prop></sync-collection>

Response:

<?xml version="1.0" encoding="utf-8" ?>
<multistatus xmlns="DAV:">
 <response>
  <href>/davical/caldav.php/richard/home/4a711f90-49e5-49d2-8b0b-c81302e03a47.ics</href>
  <propstat>
   <prop>
    <getcontenttype>text/calendar; component=vevent</getcontenttype>
    <getetag>"0d90149af381b9e7bb640a6bc013ff9e"</getetag>
   </prop>
   <status>HTTP/1.1 200 OK</status>
  </propstat>
 </response>
 <response>
  <href>/davical/caldav.php/richard/home/25266cf7-8e99-4e74-9213-31e703b930e8.ics</href>
  <propstat>
   <prop>
    <getcontenttype>text/calendar; component=vtodo</getcontenttype>
    <getetag>"73861aadb01741916ef2abac26bd1a7d"</getetag>
   </prop>
   <status>HTTP/1.1 200 OK</status>
  </propstat>
 </response>
 <response>
  <href>/davical/caldav.php/richard/home/0f197419-8762-43a0-b702-8a4a07b10845.ics</href>
  <propstat>
   <prop>
    <getcontenttype>text/calendar; component=vtodo</getcontenttype>
    <getetag>"98cf6c57df838a99c10a733c77c2c097"</getetag>
   </prop>
   <status>HTTP/1.1 200 OK</status>
  </propstat>
 </response>
 <response>
  <href>/davical/caldav.php/richard/home/a1585803-b2ee-45a0-b6f6-cc03ccf022b9.ics</href>
  <propstat>
   <prop>
    <getcontenttype>text/calendar; component=vtodo</getcontenttype>
    <getetag>"1ca23baae75fb8717e390a08e7e42a8a"</getetag>
   </prop>
   <status>HTTP/1.1 200 OK</status>
  </propstat>
 </response>
(...)

Is the format of the response supposed to look like the above?

Is the content of each item loaded (including Description/Attachments) as part of Receiving (as per Request/Response Timing)? I don't see any separate request in the DevTools Networking that show how each item data is loaded...

Description/Attachments/Attendees info loading would not need to happen at startup... if that is the case and the reason why Response is read/parsed slowly...

Hope that help.

Regards,

Flags: needinfo?(richard.leger)

Here is performance profile related to the issue reported in previous Comment 51...
https://share.firefox.dev/3V7KvH1

Here is another TB 106.0b4 (64-bit) performance profile for loading my Caldav calendar containing 5000+ items, from just after authentication till about half the number of network requests required to load it in full. With calendar.caldav.multigetBatchSize back to its default value of 100...
https://share.firefox.dev/3Mae2vA

The first request is different from the others, it's asking for a list of calendar items to ask for in the subsequent requests.

From your profile I can see that about a third of the time is spent adding items to the calendar and the UI. I know there's some optimisation that can happen there but not how much.

Most of the rest of the time is spent parsing items. I really don't know if that can be improved. Perhaps there's some overhead from parsing each item separately instead of all together, which as you mentioned works a lot faster.

In any case, I've got a lot on at the moment. I'm aware of the problems but don't have any time to do anything about them right now.

Severity: normal → S3

(In reply to Richard Leger from comment #53)

Here is another TB 106.0b4 (64-bit) performance profile for loading my Caldav calendar containing 5000+ items, from just after authentication till about half the number of network requests required to load it in full. With calendar.caldav.multigetBatchSize back to its default value of 100...
https://share.firefox.dev/3Mae2vA

Take note of bug 1572826 comment 7 if / when you update to 107.0b1.

For the record only... hoping it would help identify culprits and room for future step-by-step improvements...

Re-testing CalDav loading performance with better connectivity and:

  • Closing all network apps (especially Firefox) but Thunderbird Beta version 106.0b5 (64-bit) on Windows 10
  • One calendar with 5000+ items setup in a new profile - used for 10year+ with Thunderbird
  • Various calendar.caldav.multigetBatchSize 20/50/100(default)/300/500/1000/4000 ==> loading faster as size increases but 30seconds at best!
  • Offline Support disabled
  • No Reminders triggered during loading - Those tends to add overhead...

Such situation would happen if you work with first time calendar loading (e.g new account setup), or after disabling/renabling offline support, or with Offine Support disabled (working in online mode) at each TB startup... and would worthen as the number of calendar increases...

Calendar Week view - Unifinder Event in the next 7 days - No column sorting (default view)

Computer#1 - Desktop - Ethernet + Lease line - from within country where server is located

# TB Version Profile BatchSize LoadingTime DevTools > Networking Ping Min/Max/Avg Loss
01 106.0b5 (64-bit) https://share.firefox.dev/3exwaU5 20 3mn05s 285 Requests - 12.75MB Transferred - Finish: 3mn05s 13ms/103ms/18ms 0%
02 106.0b5 (64-bit) https://share.firefox.dev/3Txprs5 50 ~1mn50s 117 Requests - 16.66MB Transferred - Finish: 1mn50s 14ms/157ms/20ms 0%
03 106.0b5 (64-bit) https://share.firefox.dev/3exlpBa 100(default) ~1mn26s 61 Requests - 12.63MB Transferred - Finish: 3mn14s 14ms/130ms/18ms 0%
04 106.0b5 (64-bit) https://share.firefox.dev/3yNhE1n 300 ~0mn46s 24 Requests - 12.61MB Transferred - Finish: 3mn67s 13ms/192ms/21ms 0%
05 106.0b5 (64-bit) https://share.firefox.dev/3VBR1WI 500 ~0mn44s 17 Requests - 12.61MB Transferred - Finish: 1mn93s 14ms/233ms/40ms 0%
06 106.0b5 (64-bit) https://share.firefox.dev/3yMlNm3 1000 ~0mn42s 11 Requests - 12.60MB Transferred - Finish: 44s84 14ms/169ms/24ms 0%
07 106.0b5 (64-bit) https://share.firefox.dev/3yMWaRX 4000 ~0mn35s 7 Requests - 12.60MB Transferred - Finish: 2mn43s 14ms/234ms/28ms 0%

Computer#2 - Laptop - Wifi + 4G (tethering) - from abroad

# TB Version Profile BatchSize LoadingTime(Disappearance) DevTools > Networking Ping Min/Max/Avg Loss
08 106.0b5 (64-bit) https://share.firefox.dev/3VDtdSj 20 2mn99 285 Requests - 12.75MB Transferred - Finish: 2mn99s 58ms/1236ms/101ms 0%
09 106.0b5 (64-bit) https://share.firefox.dev/3EKQmMR 50 1mn50s 117 Requests - 12.66MB Transferred - Finish: 3mn40s 56ms/232ms/87ms 0%
10 106.0b5 (64-bit) https://share.firefox.dev/3EOHJkr 100(default) 1mn10s 61 Requests - 12.63MB Transferred - Finish: 2mn92s 59ms/388ms/156ms 0%
11 106.0b5 (64-bit) https://share.firefox.dev/3yPxAA1 300 0mn50s 24 Requests - 12.61MB Transferred - Finish: 2mn64s 56ms/358ms/96ms 0%
12 106.0b5 (64-bit) https://share.firefox.dev/3Tp7eNL 500 0mn46s 17 Requests - 12.61MB Transferred - Finish: 2mn90s 57ms/299ms/105ms 0%
13 106.0b5 (64-bit) https://share.firefox.dev/3VAzfmD 1000 0mn36s 11 Requests - 12.60MB Transferred - Finish: 3mn14s 56ms/260ms/91ms 0%
14 106.0b5 (64-bit) https://share.firefox.dev/3gftJ92 4000 0mn30s 7 Requests - 12.60MB Transferred - Finish: 2mn47s 59ms/325ms/187ms 0%

Set Unifinder to All Events + Title sort ascending

Computer#2 - Laptop - Wifi + 4G (tethering) - from abroad

# TB Version Profile BatchSize LoadingTime(Disappearance) DevTools > Networking Ping Min/Max/Avg Loss
15 106.0b5 (64-bit) https://share.firefox.dev/3D77hbi 100(default) 1mn81s 61 Requests - 12.63MB Transferred - Finish: 2mn43s 57ms/362ms/159ms 0%

This show that increasing batch size does improve performance drastically (x10=50% faster) though having too high batch size may have other side effects and may not necessarily work well on poor network connections... especially at 500 gaps appear between network requests...

Change of hardware and connectivity seems to have no impact on performance.

Those perf profile may help identify culprit in performance hopefully and give some comparative reference for testing future CalDAV performance improvements, if any coming up...

Richard, these are interesting results and seem to indicate a increased batch size has time saving benefits assuming someone has a connection of 4G or, perhaps, 3Mb Internet service or better. I'm testing a calendar.caldav.multigetBatchSize = 1000 right now to see if it makes any difference or not. I don't know that I have that many cal events but I'll dive in to see if my GMail accounts complain here at work or at home.

Devs, may I ask when or during what TB era was the default setting of 100 established?

Also, is there a mechanism by which TB can query a server for batch size limit or perhaps some debugging or console method that might show a threshold limit having been hit if one, say, sets it to 100,000? 10,000,000? If 100 was set arbitrarily, is there some RFC spec that states the ceiling limit? I'd suspect Google has a pretty beefy back end whereas smaller ISP CalDAV calendaring setups might balk at some really high numbers. But how to get TB to yield some quantitative metrics to potentially establish a newer baseline for 2022/2023?

For the record, after import of my Davical Calendar into Google Calendar, I re-tested and got the following results:

# TB Version Profile BatchSize LoadingTime DevTools > Networking Ping Min/Max/Avg Loss
16 106.0b5 (64-bit) https://share.firefox.dev/3CJUWYS 100(default) 1mn82s 98 Requests - 12.18MB Transferred - Finish: 1mn82s 56ms/289ms/118ms 0%
17 106.0b5 (64-bit) https://share.firefox.dev/3s5KyG0 250 1mn38 54 Requests - 12.17MB Transferred - Finish: 1mn38s 58ms/320ms/138ms 0%
18 106.0b5 (64-bit) https://share.firefox.dev/3VEl7ZA 1000 1mn37s 54 Requests - 12.17MB Transferred - Finish: 1mn37s 56ms/517ms/154mss 0%

Note:

  • My Google calendar contain fewer items because Google Calendar does not handle tasks via CalDAV protocol. So those were not imported.
  • Google Calendar (with lesser items) response may be much more verbose and of smaller batch size, therefore more numerous and therefore loading is significantly slower than with Davical Calendar (with more items) as per my previous tests.
  • Increasing calendar.caldav.multigetBatchSize above 250 has no significative impact on loading Google Calendar which somehow may have a batch limit of 250 max (either in TB protocol or at Google's end). I have no access to the OPTION request so I cannot really see where this setting may appear from.
    -- At calendar.caldav.multigetBatchSize 100 for one REPORT requesting 250 items, you get three REPORT requests of 100, 100, 50 each to load items
    -- At calendar.caldav.multigetBatchSize 250 for one REPORT request for 250 items, you get one REPORT request of 250 to load items, therefore it reduce significantly the number of requests and accelerate the loading of the Google Calendar in TB up to ~1mn38 (for a 5000+ items calendar) which is way too long even if I understand there is a network overhead :-) Especially if compared to Davical Calendar with batch size of 1000...
  • In DevTools > Networking > REPORT > Timing the WAITING time seems significant while RECEIVING time seems marginal with Google Calendar while it was the opposite with Davical Calendar in previous tests.

After running with calendar.caldav.multigetBatchSize set to 1000 for these past 2 weeks, I can't say that I see any net positive or negative effects. No complaints in Error Console or errors thrown by TB itself. For TB it seems to be business as usual. Going to reduce to 500 and test.

After a month of running calendar.caldav.multigetBatchSize set to 500, I did not encounter any errors or have any issues. Again, without some way to see if the server is having a fit over this high of a number or some other realtime manner of watching if it's helping or hurting, I can't say that I see any net positive or negative effects.

(In reply to Arthur K. (he/him) from comment #60)

After a month of running calendar.caldav.multigetBatchSize set to 500, I did not encounter any errors or have any issues. Again, without some way to see if the server is having a fit over this high of a number or some other realtime manner of watching if it's helping or hurting, I can't say that I see any net positive or negative effects.

If you are using a Gmail account please note that Google limit server side at 200... for any value of multigetBatchSize superior to 200...
so between 100 and 200 not much visible diff especially with small calendar size...
How many items in your calendar by curiosity?

(In reply to Richard Leger from comment #61)

(In reply to Arthur K. (he/him) from comment #60)

After a month of running calendar.caldav.multigetBatchSize set to 500, I did not encounter any errors or have any issues. Again, without some way to see if the server is having a fit over this high of a number or some other realtime manner of watching if it's helping or hurting, I can't say that I see any net positive or negative effects.

If you are using a Gmail account please note that Google limit server side at 200... for any value of multigetBatchSize superior to 200...
so between 100 and 200 not much visible diff especially with small calendar size...

Is it published somewhere this hard limit on GMail calendars or is it an RFC spec?

How many items in your calendar by curiosity?

I tried to figure out some easy way to get the total number of events per calendar but I can't find it easily. Is there a quick way to find out?

(In reply to Arthur K. (he/him) from comment #62)

(In reply to Richard Leger from comment #61)

(In reply to Arthur K. (he/him) from comment #60)

After a month of running calendar.caldav.multigetBatchSize set to 500, I did not encounter any errors or have any issues. Again, without some way to see if the server is having a fit over this high of a number or some other realtime manner of watching if it's helping or hurting, I can't say that I see any net positive or negative effects.

If you are using a Gmail account please note that Google limit server side at 200... for any value of multigetBatchSize superior to 200...
so between 100 and 200 not much visible diff especially with small calendar size...

Is it published somewhere this hard limit on GMail calendars or is it an RFC spec?

No RFC does set any hard limit... which would be expected.
Nor I found Google spec but you can clearly see by testing and observing Responses received from Google via TB > Dev Tools > Networking

How many items in your calendar by curiosity?
I tried to figure out some easy way to get the total number of events per calendar but I can't find it easily. Is there a quick way to find out?

You can look at content of Response in Responses received from server via TB > Dev Tools > Networking or you can use Cyberduck app (https://cyberduck.io/) to connect to the CalDAV calendar to list each event in the format of individual .ics files (it takes ~14 seconds) and in status bar it shows how many items are listed. Via WebDAV protocol.

See Also: → 1572823
Attached image 124.0b2.64-bit.png —

As you would see attached, the issue remain present in 124.0b2 (64-bit) as the default batch size is still set, per default, to 100 items per requests.

So it still takes 67 network requests and almost 2mn to load my network calendar despite improvements from Bug 1572823 and replacement of ADSL connection by a 4G one.

Blocks: 1572823

Here is a performance profile, if that can help:
https://share.firefox.dev/43207QH

Via DevTools > Networking, some requests may take 500ms of Waiting and 840ms Receiving (Parsing response), that seems a very long time for just one network request of 100 items...

Also noted, the calendar.caldav.multigetBatchSize does not appear by default with its default value in the Thunderbird > Settings > Config Editor, making it hard to know it exists for adjustment on a per user bases or via policy.

I still believe its default value should be much higher than currently or automatically adjusted based on the detected speed of connection.

I can confirm this issue. In our medium sized organisation with 150 employees we face a lot of frustration currently due to "slowness" of Thunderbird if you subscribe to more than 5-10 calendars. A lot of colleagues coordinating colleagues all over the world need more calendars in order to do their job.

We set "calendar.caldav.multigetBatchSize = 2000" and it helped a bit but only in addition to advise people to reduce their subscriptions of calendars. But this understandably annoys people.

Suggestion: It would also help to don't sync calendars which are hidden.

(In reply to benedikt.kaless from comment #66)

I can confirm this issue. In our medium sized organisation with 150 employees we face a lot of frustration currently due to "slowness" of Thunderbird if you subscribe to more than 5-10 calendars. A lot of colleagues coordinating colleagues all over the world need more calendars in order to do their job.

We set "calendar.caldav.multigetBatchSize = 2000" and it helped a bit but only in addition to advise people to reduce their subscriptions of calendars. But this understandably annoys people.

Suggestion: It would also help to don't sync calendars which are hidden.

And offline support is enabled for all of these calendars? That should improve perf since it'll only need to fetch and new changes. All clients are on 128.3.1? Network drivers fully up to date?

Due to this bug 1694709 offline-support is not activated for all those calendars because events are vanishing in a calendar using offline support.

Yes, all clients are on 128.3.1 and it is related to all kinds of clients (different windows-machines, LInux-machines)

Hi Arthur,

The point of this bug is not so much about using or not using offline support, but rather the slow speed of Thunderbird to load caldav calendar with thousand of items, as it happens every time you start TB with or without offline support and the high number of network requests and the lengthy time required to for response parsing/ui populating. While calendar perf has improved over the past year, it still not great in regards to this bug.

(In reply to Arthur K. (he/him) from comment #67)

(In reply to benedikt.kaless from comment #66)

I can confirm this issue. In our medium sized organisation with 150 employees we face a lot of frustration currently due to "slowness" of Thunderbird if you subscribe to more than 5-10 calendars.

@benedikt.kaless what do you mean by "slowness"? Is it at startup? Or afterwards during TB usage?
It is quite impressive that you can use 5-10 calendars at a time, as not long ago that would crawl TB or make it unresponsive for minutes or bring it in an unusable state.

In recent betas (currently on 132.0b4 (64-bit)) I did notice some slowness in the UI interface, delay when you type on keyboard, select recipient, click on action button, drag&drops... Etc though I cannot say if it is linked to using multiple calendars. Startup perf is also not bad as I can quickly access email... While calendars load in the backgrounds, but I have only five network calendar and only a couple with thousands of items. There three more I access less often that I keep disabled for performance reason.

And offline support is enabled for all of these calendars? That should improve perf since it'll only need to fetch and new changes.

The problem we noticed happen especially at TB startup when the calendar offline support cache is re-populated from scratch and all items from all network calendars are loaded at the same time making TB slow and unusable for a while because is all done within the same TB process (main interface, email, calendars, etc) all at the same time, "interfering" with each other.

Network drivers fully up to date?

It is not a network drivers problem it is a design problem with the calendaring infrastructure in TB.** Ideally each calendar or group of calendar shall be loaded in separate process from the main UI/email one**... So calendars would load in the background without causing overall TB performance issues. Ideally also each calendar shall be fully loaded in couple of network request max, in Web browser it only takes 1s to load a full calendar file via https, 1mn+ for the same calendar via caldav in TB. And that is just for one calendar, problem gets worth as you add more network calendars... Not the point of this bug but good to be aware of :-)

(In reply to benedikt.kaless from comment #68)

Due to this bug 1694709 offline-support is not activated for all those calendars because events are vanishing in a calendar using offline support.

I can confirm as well, though I haven't yet noticed recently in TB 128.3.1, it is too early to tell as it happens randomly. A way to fix it is do disable offline support, and re-enable it, so missing items re-appear as the full calendar is reloaded in cache. The main problem being that event disappearing/missing are not necessarily noticed by end-users as it happens silently, so it causes them to miss meetings double book them, etc... That also the reason we disabled offline support for network calendars in organisations we support as online calendar is the most reliable. Not the point of this bug but good to be aware of :-)

Regards,

(In reply to Richard Leger from comment #70)

The point of this bug is not so much about using or not using offline support, but rather the slow speed of Thunderbird to load caldav calendar with thousand of items, as it happens every time you start TB with or without offline support and the high number of network requests and the lengthy time required to for response parsing/ui populating. While calendar perf has improved over the past year, it still not great in regards to this bug.

Thanks a lot for this clarification. By the way, my impression is that old versions were faster in this behaviour. Unfortunately, I can no longer measure this, my order of magnitude is just the frustration or the number of complaints from our employees. The combination of both bugs (high number of network requests and missing events in offline support) we advise people to subscribe the personal calendar without offline support, the rest with offline support and deactivate all those regularly.

@benedikt.kaless what do you mean by "slowness"? Is it at startup? Or afterwards during TB usage?

I describe "slowness" as following: When people subscribe to more than 10 calendars thunderbird needs up to 3-5 minutes after start before it is usuable. During startup it doesn't react on mouse-clicks and doesn't load new mails. But furthermore, it responds slowly even after hours of use. For example, characters are displayed with such a delay when composing emails that entire sentences get mixed up when typing. Colleagues pre-formulate emails in text editors and copy and paste them. It's a frustrating user experience :)
On my Linux I get a warning ‘Thunderbird is not responding’ at regular intervals

It is quite impressive that you can use 5-10 calendars at a time, as not long ago that would crawl TB or make it unresponsive for minutes or bring it in an unusable state.

My impression is, it depends on the amount of events in these calendars and how often they change of course.

(In reply to benedikt.kaless from comment #68)

Due to this bug 1694709 offline-support is not activated for all those calendars because events are vanishing in a calendar using offline support.

I can confirm as well, though I haven't yet noticed recently in TB 128.3.1, it is too early to tell as it happens randomly. A way to fix it is do disable offline support, and re-enable it, so missing items re-appear as the full calendar is reloaded in cache. The main problem being that event disappearing/missing are not necessarily noticed by end-users as it happens silently, so it causes them to miss meetings double book them, etc... That also the reason we disabled offline support for network calendars in organisations we support as online calendar is the most reliable. Not the point of this bug but good to be aware of :-)

This bug unfortunately leaves users with the impression that they cannot rely on Thunderbird as a tool in their day-to-day work. If appointments are missing, they are not reminded of meetings and miss them.

The combination of both bugs is really critical for users because it means that workflows that you simply need in everyday office life don't work.
Perhaps - I am not familiar with the codebase of thunderbird - it is a quick win to don't sync hidden calendars. People are used to hide calendars via the "eye"-symbol and show calendars as soon as they want to coordinate meetings.

I think it's good that there's movement in this thread again. I can only expressly ask you to deal with this behaviour, because it's a real blocker right now.

Thanks
Benedikt

Hi,

(In reply to benedikt.kaless from comment #66)

Suggestion: It would also help to don't sync calendars which are hidden.

I think Benedikt makes a good point here, I agree with this suggestion that hidden calendar shall not be synced at startup or during Thunderbird usage is they are hidden.

While Hidden is not the same as Disabled, from a synchronisation point of view, both should stop/prevent synchronisation.

That could be an additional workaround for ongoing issues, but the bottom perf issues with calendar would remain to be resolved.

Regards,

It seems to me that increasing default calendar.caldav.multigetBatchSize should be at least a starting point.

(In reply to Arthur K. (he/him) from comment #57)

Richard, these are interesting results and seem to indicate a increased batch size has time saving benefits assuming someone has a connection of 4G or, perhaps, 3Mb Internet service or better. I'm testing a calendar.caldav.multigetBatchSize = 1000 right now to see if it makes any difference or not. I don't know that I have that many cal events but I'll dive in to see if my GMail accounts complain here at work or at home.

Devs, may I ask when or during what TB era was the default setting of 100 established?

Also, is there a mechanism by which TB can query a server for batch size limit or perhaps some debugging or console method that might show a threshold limit having been hit if one, say, sets it to 100,000? 10,000,000? If 100 was set arbitrarily, is there some RFC spec that states the ceiling limit? I'd suspect Google has a pretty beefy back end whereas smaller ISP CalDAV calendaring setups might balk at some really high numbers. But how to get TB to yield some quantitative metrics to potentially establish a newer baseline for 2022/2023?

bug 436641, in April 2010, between Thunderbird 3.0 and 3.1

Status: UNCONFIRMED → NEW
Depends on: 436641
Ever confirmed: true
Summary: Lightning network REPORT request receiving time too high (0.5 to 3 seconds per request) - Loading one ~4000 events network calendar takes more than 1 minute - 46 network requests required (isn't it too much?) → REPORT request receiving time too high (0.5 to 3 seconds per request) - Loading one ~4000 events network calendar takes more than 1 minute - 46 network requests required (isn't it too much?)

(In reply to Wayne Mery (:wsmwk) from comment #73)

It seems to me that increasing default calendar.caldav.multigetBatchSize should be at least a starting point.

This setting shall also

  • appears by default in the Global Editors as Preferences with its default value, it should not be hidden/hardcoded in the code.
  • be adjustable by end users
  • be made accessible via policies

This way end users and organisation could adjust its value as required from their end by any means they see better fit best their needs.

A max limit may need to be set to avoid situation like bug 436641.

Also as observed in Comment 58 testing, increasing the value may not necessarily have the expected impact if suppliers like Google may have limit of their own at their end.

(In reply to benedikt.kaless from comment #71)

(In reply to benedikt.kaless from comment #68)

Due to this bug 1694709 offline-support is not activated for all those calendars because events are vanishing in a calendar using offline
This bug unfortunately leaves users with the impression that they cannot rely on Thunderbird as a tool in their day-to-day work. If appointments are missing, they are not reminded of meetings and miss them.
The combination of both bugs is really critical for users because it means that workflows that you simply need in everyday office life don't work.

Isn't there a bug for parity with Outlook or Exchange compatibility? Should this one be marked as blocking that bug?

Isn't there a bug for parity with Outlook or Exchange compatibility? Should this one be marked as blocking that bug?

Yes, but this bug is not about a missing feature, so it wouldn't block.

Summary: REPORT request receiving time too high (0.5 to 3 seconds per request) - Loading one ~4000 events network calendar takes more than 1 minute - 46 network requests required (isn't it too much?) → REPORT request receiving time too high (0.5 to 3 seconds per request). Loading one ~4000 events network calendar, both online and offline, more than 1 minute and 46 network requests. Adjust calendar.caldav.multigetBatchSize preference?

(In reply to Richard Leger from comment #74)

(In reply to Wayne Mery (:wsmwk) from comment #73)

It seems to me that increasing default calendar.caldav.multigetBatchSize should be at least a starting point.

This setting shall also ...

Not sure we need to expose it if a more modern default is appropriate.

The size of 100 was determined 15 years ago based on bug 436641 comment 5. "simon_at_orcl noted that most clients split the requests into batches of 100".

And calendars are much bigger now than 15 years ago.

Anyone using a value >100 in production?
What do other clients do today?

OS: Unspecified → All
See Also: → 759685, 1267077

I set this value to 4000.
To be honest: DavX5 seems to have an implementation which works quite well.
This approach seems interessting as well and could solve this networking issue, isn't it?

(In reply to Wayne Mery (:wsmwk) from comment #77)

(In reply to Richard Leger from comment #74)

(In reply to Wayne Mery (:wsmwk) from comment #73)

It seems to me that increasing default calendar.caldav.multigetBatchSize should be at least a starting point.

This setting shall also ...

Not sure we need to expose it if a more modern default is appropriate.

The size of 100 was determined 15 years ago based on bug 436641 comment 5. "simon_at_orcl noted that most clients split the requests into batches of 100".

And calendars are much bigger now than 15 years ago.

Anyone using a value >100 in production?
What do other clients do today?

Forgive me if this was answered somewhere or even here but is there an official RFC spec that dictates the request value or is it arbitrary? The 1st comment of bug 435541 has a link to an apple.com reference but it's a dead and I only found RFC4791 (https://www.ietf.org/rfc/rfc4791.txt). A cursory perusal of the spec doesn't overtly dictate a minimum or maximum chunk size of how many items to download per request. I suppose it's value could be dictated by the server app serving the calendaring? Has anyone tried setting the value to something like 1,000,000,000 and seen if any error gets thrown?

(In reply to Arthur K. (he/him) from comment #79)

I suppose it's value could be dictated by the server app serving the calendaring? Has anyone tried setting the value to something like 1,000,000,000 and seen if any error gets thrown?

As per previous tests and results from Comment 56 and Comment 58, above 4000 there weren't much improvement in speed as other limitations kicked in at the time. Also Google was limiting to 250 server side, at the time, so a big value had little effect at the loading time of Calendar for that service provider but then that become a provider issue not a Thunderbird issue.
CalDAV being very verbose xml protocol, parsing and processing data by Thunderbird may also have played an additional role in loading delay at the time of testing, but that is not the point of this bug.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: