997784 - Modifying invitation attendance causes Google to send out notifications

Status: UNCONFIRMED

(Calendar :: Provider: CalDAV, defect)

Description

[Jan Henning [:JanH]]

Attachment: Google Calendar notifications test.xlsx

First a few words about my setup: I have a Google account with my regular private e-mail address (non-Google, so Google definitively knows about my events only through Lightning) as the primary address. The calendar belonging to that account is accessed in Thunderbird through CalDAV, with offline support activated. As far as advanced calendar settings are concerned, to my knowledge I've only got calendar.google.enableEmailInvitations set to true.

Steps to reproduce:
Create a second Thunderbird profile (launch the second Thunderbird instance with the -no-remote switch, so you can easily switch back and forth between the two profiles), preferably using a different e-mail address, so things in your inbox don't get too confusing and you avoid any potential additional bugs caused by inviting yourself ;-) Configure this profile to use a locally stored calendar.
Then try one or several of the tests from the attached Excel sheet, e.g. Test # 5:

1. In the second Thunderbird profile, create a new event, and add your main account to the list of invitees. After saving the event, confirm the dialogue asking you whether to send out a notification.
2. You should now receive the invitation in your main profile. Decline it, and deny Thunderbird sending back a confirmation. The event should now appear in your main calendar - both locally and online in Google's calendar (or on any synced devices). Your second profile shouldn't receive any e-mail at this stage.
3. In your main profile, change the event's status to Accept and again deny when asked to send a notification. In your second profile, you should now receive a notification from Google. If you had accepted Lightning's offer to send a notification, you would actually receive two e-mails at this point - one sent by Lightning, and the other by Google.
4. In your second profile, delete the event and send the notification.
5. In your main profile, accept the cancelling of the event. No e-mail will be received in your second profile.

Basically, it looks like you can add and delete invitations from a Google calendar without Google sending out any potentially unwanted/duplicated notifications.
However if you later decide to modify the attendance status, Google will send out a notification back to the organiser whether you want it or not.

If - in your main profile - you delete the event, and then add it back from the invitation e-mail, Google will somehow recognise that the event isn't really "new" and consequently send back a notification as well.

Although to be honest, as apparently Google has had a history of other, similar bugs, I'm not really sure whether this is really some bug in Lightning's implementation, or rather one in Google's API.

Comment 1

[[:MakeMyDay]]

Thank you for your report and elaborating the test cases.

Basically, this seems to be an issue on Google's end. The Outlook compatibility dialog cannot appear, if there may have happened any Lightning triggered calDAV scheduling. As you reported, the dialog poped up - so it must be Google deciding to send out the status update.

As I personally don't use Google Calendar, you maybe want to enable debug logging (set prefs calendar.debug.log and calendar.debug.log.verbose to true - if no such prefs are available, just create them) and provide the log after executing one of the test cases in question to make sure it's not a Lightning issue.


Two further notes to your report:

Stop sending out a cancellation if you denied sending with the popup before as described in no. 2 of #comment1 is already fixed with bug 966292.

The pref calendar.google.enableEmailInvitations is for the Google provider only and has no effect when using caldav.

Comment 2

[Jan Henning [:JanH]]

Attachment: Log test case #5 - main profile.txt

Comment 3

[Jan Henning [:JanH]]

I've added the log for test case #5.

One thing I'm wondering about is that at least in the log, Lightning doesn't seem to send out the "SCHEDULE-AGENT" parameter, even in those cases where I grant it the permission to send out the notification (not covered in the above log).

While that doesn't explain why Google would send out notifications only when modifying an existing event, or alternatively re-creating a formerly deleted event, it might nevertheless have some influence on the issue.

Comment 4

[[:MakeMyDay]]

In general, the caldav implementation is pretty old and is based on a former work in progress version of the RfC. It realy needs an update - hopefully this will happen this summer with a GSoC project.

Two further questions:

Does the observed behaviour also occur if offline support is disabled (for the regular profile in your setup for tc#5)?

Have you the ability to test this also with a current nightly?

> One thing I'm wondering about is that at least in the log, Lightning doesn't
> seem to send out the "SCHEDULE-AGENT" parameter, even in those cases where I
> grant it the permission to send out the notification (not covered in the
> above log).
> 
> While that doesn't explain why Google would send out notifications only when
> modifying an existing event, or alternatively re-creating a formerly deleted
> event, it might nevertheless have some influence on the issue.

It may explain the issue, because the initial response is the first time, the attendees's server get the event (and it get's the itip message as you see in the log), while on later modifications (where it get's the modified ical) the server then acts comforming to RfC to interprate the missing SCHEDULE-AGENT as if it's set to SERVER and sends out a notification.

Comment 5

[Jan Henning [:JanH]]

> Does the observed behaviour also occur if offline support is disabled (for
> the regular profile in your setup for tc#5)?

Yes.

> Have you the ability to test this also with a current nightly?

I did, and there wasn't any change in behaviour.

I did take a look at Lightning's source though. The relevant bit seems to be contained in modules/calItipUtils.jsm at line 1046.
Apparently SCHEDULE-AGENT=CLIENT is only sent out if the server claims to be capable of scheduling, so this means that apparently Google's CalDAV API isn't advertising its capabilities correctly.
I've fudged that check to alway return true, and subsequently upon adding the event from the invitation into the calendar, the relevant line from the log looked like this:

ORGANIZER;RSVP=TRUE;CN=Jan Henning;PARTSTAT=ACCEPTED;ROLE=CHAIR;SCHEDULE-A
 GENT=CLIENT:mailto:[e-mail]

However if I change the attendance status afterwards, the SCHEDULE-AGENT parameter once again doesn't get send out.
I wasn't able to find the place where the updated message gets created and passed to CalDAV (so I could fudge in the SCHEDULE-AGENT parameter in order to see whether it would make any difference), however using test case #7 (deleting the event in my main profile and the adding it back) it seems like Google is actually ignoring that parameter.
When I add the formerly deleted event back into my main calendar, Lightning once again includes the SCHEDULE-AGENT=CLIENT parameter, however Google sends out a notification anyway.

So Lightning's handling of the SCHEDULE-AGENT=CLIENT parameter might or might not be entirely correct, but unless somebody spots something I've overlooked, it seems like the main fault is on Google's side. I guess I have to try filing a bug with them...

Comment 6

[[:MakeMyDay]]

(In reply to JanH from comment #5)

> I did take a look at Lightning's source though. The relevant bit seems to be
> contained in modules/calItipUtils.jsm at line 1046.
> Apparently SCHEDULE-AGENT=CLIENT is only sent out if the server claims to be
> capable of scheduling, so this means that apparently Google's CalDAV API
> isn't advertising its capabilities correctly.

I don't know if this is still the case, but didn't so in former days - there is already a place to spoof the result of the OPTIONS request on Google calandar server at http://mxr.mozilla.org/comm-central/source/calendar/providers/caldav/calDavCalendar.js#1888 - to inject thisCalendar.hasAutoScheduling=true there would ensure to have the SCHEDULE-AGENT=CLIENT param set on incoming invitations.

> I've fudged that check to alway return true, and subsequently upon adding
> the event from the invitation into the calendar, the relevant line from the
> log looked like this:
> 
> ORGANIZER;RSVP=TRUE;CN=Jan Henning;PARTSTAT=ACCEPTED;ROLE=CHAIR;SCHEDULE-A
>  GENT=CLIENT:mailto:[e-mail]
> 
> However if I change the attendance status afterwards, the SCHEDULE-AGENT
> parameter once again doesn't get send out.

The problem is here, that currently there are several places in code, were attendee properties were dropped and recreated while processing. We plan to get rid of it and probably with bug 459683 fixed, it done for the most part.

> I wasn't able to find the place where the updated message gets created and
> passed to CalDAV (so I could fudge in the SCHEDULE-AGENT parameter in order
> to see whether it would make any difference), however using test case #7
> (deleting the event in my main profile and the adding it back) it seems like
> Google is actually ignoring that parameter.

For testing, whether Google behaves correctly if the SCHEDULE-AGENT param is always set, you may have a look at http://mxr.mozilla.org/comm-central/source/calendar/providers/caldav/calDavCalendar.js#2645 - this is the last resort before http sending - event though this is probably not the best place for a later fix. The best to intercept is probably right before line 2679.

> When I add the formerly deleted event back into my main calendar, Lightning
> once again includes the SCHEDULE-AGENT=CLIENT parameter, however Google
> sends out a notification anyway.

This is an edge case (and you only can expirience this, because Lightning does not delete scheduling mails (see bug 397322) after processing like other clients do) - but provided the SCHEDULE-AGENT param would resolve the issue in general, this also would no longer occur, because the copy on the server containes the params.

> So Lightning's handling of the SCHEDULE-AGENT=CLIENT parameter might or
> might not be entirely correct, but unless somebody spots something I've
> overlooked, it seems like the main fault is on Google's side. I guess I have
> to try filing a bug with them...

Probably you may want to test the injection of the param before sending first, but if this doesn't resolve the issue I agree to consider this a Google bug.

Comment 7

[Jan Henning [:JanH]]

(In reply to MakeMyDay from comment #6)
> I don't know if this is still the case, but didn't so in former days - there
> is already a place to spoof the result of the OPTIONS request on Google
> calandar server at
> http://mxr.mozilla.org/comm-central/source/calendar/providers/caldav/
> calDavCalendar.js#1888 - to inject thisCalendar.hasAutoScheduling=true there
> would ensure to have the SCHEDULE-AGENT=CLIENT param set on incoming
> invitations.

Thanks for the hint. Interestingly, setting hasAutoScheduling=true for Google calendars leads to some contradicting behaviour in my opinion:
On the one hand as you pointed out it sets SCHEDULE-AGENT=CLIENT param on incoming invitations, thereby as far as I understand it indicating to the server that Thunderbird will handle responding to invitations, on the other hand it seems like the option to send notifications from Thunderbird gets disabled, because the confirmation dialogue doesn't pop up any more and Thunderbird doesn't send out any replies any more.

So if the DAV server would actually honour the SCHEDULE-AGENT parameter, no message would get sent out at all:
The server wouldn't send anything because it believes the client is going to do it, while some other piece of Lightining's code believes that server will be responsible for sending back notifications if the calendar supports auto-scheduling.

> For testing, whether Google behaves correctly if the SCHEDULE-AGENT param is
> always set, you may have a look at
> http://mxr.mozilla.org/comm-central/source/calendar/providers/caldav/
> calDavCalendar.js#2645 - this is the last resort before http sending - event
> though this is probably not the best place for a later fix. The best to
> intercept is probably right before line 2679.

At least in my setup, that bit of code never got called at all when modifying an existing event. By searching for the corresponding log message, I found the section around line 720 instead, and added this bit of code after var thisCalendar = this;

if (aNewItem.organizer) {
    aNewItem.organizer.setProperty("SCHEDULE-AGENT","CLIENT");
}

This did add that parameter when modifying existing events, however in any case it didn't change Google's behaviour.

Comment 8

[[:MakeMyDay]]

Touching this code seems like opening Pandora's box ;-)

From what I've seen in Google API description, they don't care about the SCHEDULE-AGENT property and rely on an additional URI param of the event url (see https://developers.google.com/google-apps/calendar/v3/reference/events/update) - though their default behaviour is not to send out notifications. That said, I guess it don't make sense to go ahead regarding the SCHEDULE-AGENT param here. But maybe Philipp has any ideas on this.

Btw: There is now a GSoC project to update the scheduling code to current standards. To follow any progress there, see bug 1000390.

Comment 9

[Philipp Kewisch [:Fallen][📱📆]]

Note that the v3 API is not the caldav api, but they might handle this the same internally. I agree this is a Pandora's box, I really would like to see a solid set of tests before we touch this. We could probably do a whole GSoC on invitations test alone, so I wouldn't mind you supporting him in writing those tests. Of course we can't really emulate the server (unless we write a whole caldav fakeserver, which is a bigger projects on its own), but maybe we can write mozmill or unit tests that make sure the SCHEDULE-AGENT behavior is consistent.