Bugzilla

(In reply to Magnus Melin [:mkmelin] from comment #73)
> You expect Thunderbird to process multi-megabytes of calendar data super fast.
> The network lag alone is going to give you bad UX.
> An email is usually in < 100KB, so then it doesn't matter. If you need to load many large calendars maybe 1000 larger than the mail, and display the data fast: that *is* cache is is for.

Please note that people loading their calendar for the first time will be affected by this bug!

Yes cache mode (Offline Support) allow to work offline... 
Yes cache mode is helping to prevent too much network access and compensate network lags...
... but let me be clear ones more here, it is is **not** the problem at stake nor **the fix** for this bug... 

So let's just disregard cache mode for now shall we?

I cannot help but feel that I am misunderstood (and other end-users encountering issues out there - not just me).

I will try to draw the picture differently, lets concentrate about speed of loading the same CalDav calendar (4000+ items) with Offline Support disabled in a new blank profile in various version of TB, that something you may be able to relate to:

|TB Version|Loading Time|DevTools > Networking|
|:--|:--|:--|
|60.8.0 (32-bit) ESR|~100mn+ (with sorting)|48 requests - 7.98MB transferred|
|60.8.0 (32-bit) ESR|~6mn23s (without sorting)|idem|
|70.0b2 (64-bit)|**~1mn02s**|49 Requests - 8.25MB Transferred|
|78.0b1 (64-bit)|~10mn30s|50 Requests - 9.25MB Transferred|

Am I really the only one seeing those results?

As you can see clearly (I hope), the fastest measured **~1mn02s** was obtained so far with *TB 70.0b2*  with the same calendar over an ADSL line.
(they might have been slightly better than this but cannot recall stats... but you get the picture!) 

This result was accomplished thanks to work done in Bug 1502923 and few follow up bugs... but that had been quite a fight at the time and lot time spent to get the result! Since then I have kept statistics on regular bases in one of the follow up Bug 1572823.

The only difference between tests above is the TB version, the rest is the same.
Similar results obtained when TB running on the same LAN as the server or a 4G network! So network is **not** the issue here :-)

Clearly this should be sufficient evidence that great performance results can be obtained with TB CalDav feature, **without** using cache mode (which would just be the cherry on the cake!).
If the dev team only test with Offline Support enabled (the default) they might not see issues during the development cycle linked to online mode! They also need to test the online mode on large calendars... and not advertise cache mode as the fix for performance issue... which is not! Just hiding dust under the carpet to me :-)

To go event further, reach a target of below 1mn which is likely achievable, you would then need to work on the number of network requests (see dedicated Bug 1572823) which seems way too numerous, I believe that network request could be reduced to 4 if the batch were requested in 1000 items instead of 100 currently... 

My entire .ics calendar is ~2MB in size and takes 1-2 sec to load entirely via http! Yes CalDav XML protocol create a lot of overhead so size of data transferred is increased by about x3 time... but that is not an issue for now... this can be dealt with later perhaps... network and size of data transfer is *not* the issue here.

In summary, here are the few major issues currently affecting TB CalDav in online mode as I see it:

- Gap between pics of network request/response ````/\_/\__/\___```` which correspond to the time TB takes to process the response received and slowly load items in the various views (UX) before sending next request, delay often increasing over time. At best the network traffic should look like this ````/\/\/\/\```` meaning no processing delay by TB (no gap) between two network request/response (one pic = one request+response). Attachment 9159427 [details] published in Comment 48 shall give you a rough idea of what I mean it clearly show deterioration... also more network graph attachment available in Bug 1502923 and Bug 1572823 for various TB version.

- The number of network requests sent by TB to load the calendar - 50 requests really?! This is way too much to my opinion... still to be dealt with in separate dedicated bug Bug 1543953. 
There may be additional way of improving request/response by not preloading all the data at first  perhaps but only the essentials... especially when working online... I only need to load body and attachment when I click on item to edit it/view it... but that is another story entirely.

- The dev team not seeing such issues (for some unknown reason) or not testing for it perhaps, to allow them to fix and prevent regression over iteration... as said previously each iteration cycle shall bring us closer to better performance with CalDav... at least between ESR version... the fact the such regression slipped between the finger seems to suggest that the testing suite of CalDav feature during development may need to be reviewed somehow...

- Performance deteriorating fast as number of enabled CalDAV calendar increases...

- UX performance itself to load senselessly items in the various views...

Only the first point in that list is to be dealt with here.
I am not sure how I can explain better or help better than the above.
                 
Don't get me wrong I love TB and the team is doing a great job just trying to raise concern about a performance culprit observed and experienced for way too long (prior the new governance) with hope it may help to fix long term and before next ESR :-)

Regards,

(In reply to Magnus Melin [:mkmelin] from comment #73)
> You expect Thunderbird to process multi-megabytes of calendar data super fast.
> The network lag alone is going to give you bad UX.
> An email is usually in < 100KB, so then it doesn't matter. If you need to load many large calendars maybe 1000 larger than the mail, and display the data fast: that *is* cache is is for.

Please note that people loading their calendar for the first time will be affected by this bug!

Yes cache mode (Offline Support) allow to work offline... 
Yes cache mode is helping to prevent too much network access and compensate network lags...
... but let me be clear ones more here, it is is **not** the problem at stake nor **the fix** for this bug... 

So let's just disregard cache mode for now shall we?

I cannot help but feel that I am misunderstood (and other end-users encountering issues out there - not just me).

I will try to draw the picture differently, lets concentrate about speed of loading the same CalDav calendar (4000+ items) with Offline Support disabled in a new blank profile in various version of TB, that something you may be able to relate to:

|TB Version|Loading Time|DevTools > Networking|
|:--|:--|:--|
|60.8.0 (32-bit) ESR|~100mn+ (with sorting)|48 requests - 7.98MB transferred|
|60.8.0 (32-bit) ESR|~6mn23s (without sorting)|idem|
|70.0b2 (64-bit)|**~1mn02s**|49 Requests - 8.25MB Transferred|
|78.0b1 (64-bit)|~10mn30s|50 Requests - 9.25MB Transferred|

Am I really the only one seeing those results?

As you can see clearly (I hope), the fastest measured **~1mn02s** was obtained so far with *TB 70.0b2*  with the same calendar over an ADSL line.
(they might have been slightly better than this but cannot recall stats... but you get the picture!) 

This result was accomplished thanks to work done in Bug 1502923 and few follow up bugs... but that had been quite a fight at the time and lot time spent to get the result! Since then I have kept statistics on regular bases in one of the follow up Bug 1572823.

The only difference between tests above is the TB version, the rest is the same.
Similar results obtained when TB running on the same LAN as the server or a 4G network! So network is **not** the issue here :-)

Clearly this should be sufficient evidence that great performance results can be obtained with TB CalDav feature, **without** using cache mode (which would just be the cherry on the cake!).
If the dev team only test with Offline Support enabled (the default) they might not see issues during the development cycle linked to online mode! They also need to test the online mode on large calendars... and not advertise cache mode as the fix for performance issue... which is not! Just hiding dust under the carpet to me :-)

To go even further, reach a target of below 1mn which is likely achievable, you would then need to work on the number of network requests (see dedicated Bug 1572823) which seems way too numerous, I believe that network request could be reduced to 4 if the batch were requested in 1000 items instead of 100 currently... 

My entire .ics calendar is ~2MB in size and takes 1-2 sec to load entirely via http! Yes CalDav XML protocol create a lot of overhead so size of data transferred is increased by about x3 time... but that is not an issue for now... this can be dealt with later perhaps... network and size of data transfer is *not* the issue here.

In summary, here are the few major issues currently affecting TB CalDav in online mode as I see it:

- Gap between pics of network request/response ````/\_/\__/\___```` which correspond to the time TB takes to process the response received and slowly load items in the various views (UX) before sending next request, delay often increasing over time. At best the network traffic should look like this ````/\/\/\/\```` meaning no processing delay by TB (no gap) between two network request/response (one pic = one request+response). Attachment 9159427 [details] published in Comment 48 shall give you a rough idea of what I mean it clearly show deterioration... also more network graph attachment available in Bug 1502923 and Bug 1572823 for various TB version.

- The number of network requests sent by TB to load the calendar - 50 requests really?! This is way too much to my opinion... still to be dealt with in separate dedicated bug Bug 1543953. 
There may be additional way of improving request/response by not preloading all the data at first  perhaps but only the essentials... especially when working online... I only need to load body and attachment when I click on item to edit it/view it... but that is another story entirely.

- The dev team not seeing such issues (for some unknown reason) or not testing for it perhaps, to allow them to fix and prevent regression over iteration... as said previously each iteration cycle shall bring us closer to better performance with CalDav... at least between ESR version... the fact the such regression slipped between the finger seems to suggest that the testing suite of CalDav feature during development may need to be reviewed somehow...

- Performance deteriorating fast as number of enabled CalDAV calendar increases...

- UX performance itself to load senselessly items in the various views...

Only the first point in that list is to be dealt with here.
I am not sure how I can explain better or help better than the above.
                 
Don't get me wrong I love TB and the team is doing a great job just trying to raise concern about a performance culprit observed and experienced for way too long (prior the new governance) with hope it may help to fix long term and before next ESR :-)

Regards,

(In reply to Magnus Melin [:mkmelin] from comment #73)
> You expect Thunderbird to process multi-megabytes of calendar data super fast.
> The network lag alone is going to give you bad UX.
> An email is usually in < 100KB, so then it doesn't matter. If you need to load many large calendars maybe 1000 larger than the mail, and display the data fast: that *is* cache is is for.

Please note that people loading their calendar for the first time will be affected by this bug!

Yes cache mode (Offline Support) allow to work offline... 
Yes cache mode is helping to prevent too much network access and compensate network lags...
... but let me be clear ones more here, it is is **not** the problem at stake nor **the fix** for this bug... 

So let's just disregard cache mode for now shall we?

I cannot help but feel that I am misunderstood (and other end-users encountering issues out there - not just me).

I will try to draw the picture differently, lets concentrate about speed of loading the same CalDav calendar (4000+ items) with Offline Support disabled in a new blank profile in various version of TB, that something you may be able to relate to:

|TB Version|Loading Time|DevTools > Networking|
|:--|:--|:--|
|60.8.0 (32-bit) ESR|~100mn+ (with sorting)|48 requests - 7.98MB transferred|
|60.8.0 (32-bit) ESR|~6mn23s (without sorting)|idem|
|70.0b2 (64-bit)|**~1mn02s**|49 Requests - 8.25MB Transferred|
|78.0b1 (64-bit)|~10mn30s|50 Requests - 9.25MB Transferred|

Am I really the only one seeing those results?

As you can see clearly (I hope), the fastest measured **~1mn02s** was obtained so far with *TB 70.0b2*  with the same calendar over an ADSL line.
(they might have been slightly better than this but cannot recall stats... but you get the picture!) 

This result was accomplished thanks to work done in Bug 1502923 and few follow up bugs... but that had been quite a fight at the time and lot time spent to get the result! Since then I have kept statistics on regular bases in one of the follow up Bug 1572823.

The only difference between tests above is the TB version, the rest is the same.
Similar results obtained when TB running on the same LAN as the server or a 4G network! So network is **not** the issue here :-)

Clearly this should be sufficient evidence that great performance results can be obtained with TB CalDav feature, **without** using cache mode (which would just be the cherry on the cake!).
If the dev team only test with Offline Support enabled (the default) they might not see issues during the development cycle linked to online mode! They also need to test the online mode on large calendars... and not advertise cache mode as the fix for performance issue... which is not! Just hiding dust under the carpet to me :-)

To go even further, reach a target of below 1mn which is likely achievable, you would then need to work on the number of network requests (see dedicated Bug 1572823) which seems way too numerous, I believe that network request could be reduced to 4 if the batch were requested in 1000 items instead of 100 currently... 

My entire .ics calendar is ~2MB in size and takes 1-2 sec to load entirely via http! Yes CalDav XML protocol create a lot of overhead so size of data transferred is increased by about x3 time... but that is not an issue for now... this can be dealt with later perhaps... network and size of data transfer is *not* the issue here.

In summary, here are the few major issues currently affecting TB CalDav in online mode as I see it:

- Gap between pics of network request/response ````/\_/\__/\___```` which correspond to the time TB takes to process the response received and slowly load items in the various views (UX) before sending next request, delay often increasing over time. At best the network traffic should look like this ````/\/\/\/\```` meaning no processing delay by TB (no gap) between two network request/response (one pic = one request+response). Attachment 9159427 [details] published in Comment 48 shall give you a rough idea of what I mean it clearly show deterioration... also more network graph attachment available in Bug 1502923 and Bug 1572823 for various TB version.

- The number of network requests sent by TB to load the calendar - 50 requests really?! This is way too much to my opinion... still to be dealt with in separate dedicated bug Bug 1543953. 
There may be additional way of improving request/response by not preloading all the data at first  perhaps but only the essentials... especially when working online... I only need to load body and attachment when I click on item to edit it/view it... but that is another story entirely.

- The dev team not seeing such issues (for some unknown reason) or not testing for it perhaps, to allow them to fix and prevent regression over iteration... as said previously each iteration cycle shall bring us closer to better performance with CalDav... at least between ESR version... the fact the such regression slipped between the finger seems to suggest that the testing suite of CalDav feature during development may need to be reviewed somehow...

- Performance deteriorating fast as number of enabled CalDAV calendar increases...

- UX performance itself to load seemlessly items in the various views...

Only the first point in that list is to be dealt with here.
I am not sure how I can explain better or help better than the above.
                 
Don't get me wrong I love TB and the team is doing a great job just trying to raise concern about a performance culprit observed and experienced for way too long (prior the new governance) with hope it may help to fix long term and before next ESR :-)

Regards,