534228 - Clicking "Update" on an ICS event invitation reply results in MODIFICATION_FAILED on Sun Calendar Server 7 with CalDAV

Status: NEW

(Calendar :: Provider: CalDAV, defect)

Description

[Jdelic]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091204 Lightning/1.0b2pre Thunderbird/3.0

I'm using today's nightly build.

When I create an event, I can send out email invitations to other attendees, those are sent out fine. An attendee using the same setup (TB 3.0 final, Lightning 20091210) can accept the event and the reply is sent out fine. When I get the reply with the acceptance message and click the "Update"-button, this results in a MODIFICATION_FAILED error.

In the server logs I can only see a HTTP PUT-Request with no associated error message.

The error console shows:
Warning: There has been an error reading data for calendar: Test Calendar.  However, this error is believed to be minor, so the program will attempt to continue. Error code: DAV_PUT_ERROR. Description: There was an error storing the item on the server.

Error: An error occurred when writing to the calendar Test Calendar! Error code: MODIFICATION_FAILED. Description: 
Source File: file:///C:/Users/jm/AppData/Roaming/Thunderbird/Profiles/em0iq4cc.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/modules/calUtils.jsm -> file:///C:/Users/jm/AppData/Roaming/Thunderbird/Profiles/em0iq4cc.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calCalendarManager.js
Line: 1000

Reproducible: Always

Comment 1

[Stefan Sitter [:ssitter]]

Could you clarify the calendar server in use? I always thought the Sun Calendar Server uses the WCAP protocol provider. But your summary mentions the CalDAV protocol provider and the bug was filed for the ICS provider.

Comment 2

[Jdelic]

Sun has a new calendar server product that is a complete rewrite of the old server. Calendar Server 6.3 is currently still supported but will not be extended. That is the one that used WCAP. Calendar Server 7 is based around CalDAV and completely incompatible to the old one. Thanks to the marketing genius that is Sun Microsystems this difference is going to haunt a lot of people for a long time ;-).

More information can be found here: http://wikis.sun.com/display/CommSuite7/Home

Anyway, Sun claims full compatibility with Thunderbird/Lightning and as far as I can see no error actually occurs on the server.

I filed the bug against ICS/WebDAV because I thought that this component likely implements the .ics attachment handling. If ICS/WebDAV refers to .ics-files published server-side only, please feel free to reassign this bug wherever it actually belongs. I'm completely new to Mozilla Calendar's module structure.

Comment 3

[Jdelic]

Oh, after some more research I could imagine that the behavior I'm seeing could be related to Bug 456706. 

As that bug was filed against "Provider: CalDAV", I'll change this one here, too. I'm sorry for any confusion that caused.

Comment 4

[Jdelic]

I'm really sorry for all the spam I'm causing. Enabling the verbose error log, I get the following additional info:

CalDAV: Unexpected status on modifying item on Test Calendar: 403

However, unlike Bug 456706 the change does not persist on the server-side.

Comment 5

[Bruno Browning]

A 403 is an error response by the server meaning that the request was a legal request, but the server is refusing to respond to it. You might find more information in the server logs.

Comment 6

[Jdelic]

ok, I found the 403 response indicated in the access log, so I attached a WireShark session to it. Here is what I got (Email addresses and user data removed by me, but it was valid):

Request:

PUT /dav/home/jm.bugtracking/calendar/2c67aa43-247e-452f-8272-af2c4fe0a4bc.ics HTTP/1.1
Host: calendar
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091204 Lightning/1.0b2pre Thunderbird/3.0
Accept: text/xml
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: utf-8,*;q=0.1
Keep-Alive: 300
Connection: keep-alive
Content-Length: 999
Content-Type: text/calendar; charset=utf-8
If-Match: "1260555697000.1"
Authorization: Basic *********REDACTED***********
Pragma: no-cache
Cache-Control: no-cache

BEGIN:VCALENDAR
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Europe/Paris
X-LIC-LOCATION:Europe/Paris
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
CREATED:20091211T182113Z
LAST-MODIFIED:20091211T185957Z
DTSTAMP:20091211T185957Z
UID:2c67aa43-247e-452f-8272-af2c4fe0a4bc
SUMMARY:Test Event
ORGANIZER;RSVP=TRUE;PARTSTAT=ACCEPTED;ROLE=CHAIR:mailto:*********REDACTED***********
ATTENDEE;RSVP=TRUE;PARTSTAT=ACCEPTED;ROLE=REQ-PARTICIPANT;RECEIVED-SEQUENC
 E=0;RECEIVED-DTSTAMP=20091211T182241Z:mailto:*********REDACTED***********
DTSTART;TZID=Europe/Paris:20091215T200000
DTEND;TZID=Europe/Paris:20091215T210000
X-MOZ-SEND-INVITATIONS:TRUE
X-MOZ-GENERATION:1
END:VEVENT
END:VCALENDAR


And this makes the server answer with this:

HTTP/1.1 403 Forbidden
Content-Type: application/xml;charset="utf-8"
Transfer-Encoding: chunked
Date: Fri, 11 Dec 2009 19:00:08 GMT

<?xml version="1.0" encoding="UTF-8"?><D:error xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav">
<C:valid-calendar-data/></D:error>

This should make it possible at least to determine whether this should be a bug filed against the server or against Lightning... Any ideas?

Comment 7

[Jdelic]

Ok, more Spam, but I have an update. Setting the log level to FINE, I got the following error: 

FINE    [2009-12-12T00:00:00.818+0100] <...DavServlet.service> Got a non standard condition: failed to parse - Error at line 30: Invalid parameter name: RECEIVED-SEQUENCE
FINE    [2009-12-12T00:00:00.818+0100] <...DavServlet.service> Caused by: Error at line 30: Invalid parameter name: RECEIVED-SEQUENCE

hmm... is there anything I can do about that?

Comment 8

[Jdelic]

Ok.. I debugged this further. Accepting invitations from an email message does not work with Sun Calendar Server 7, because Lightning sends two fields that CS7 does not understand:

  * RECEIVED-DTSTAMP and
  * RECEIVED-SEQUENCE

which makes it fail with an error instead of updating the calendar entry.

Removing those two fields manually from a iCalendar-payload captured through WireShark and subsequently sending them to the server using curl actually works and results in a "204 No Content" response and an updated calendar entry.

Modifying the above request body we get:

[...]
ORGANIZER;RSVP=TRUE;PARTSTAT=ACCEPTED;
 ROLE=CHAIR:mailto:*********REDACTED***********
ATTENDEE;RSVP=TRUE;PARTSTAT=ACCEPTED;
 ROLE=REQPARTICIPANT:mailto:*********REDACTED***********
DTSTART;TZID=Europe/Paris:20091215T200000
[...]

which then works as expected. Can this be changed/fixed in Lightning's code in a way that doesn't break other things?

Comment 9

[Bruno Browning]

Thanks for the sleuthing. Our current caldav-sched code is pretty much written against version 04 of the spec, which specified those two fields. They were removed in version 05 and are not present in the current version (08). We should remove the code that implements these two fields, even though doing so may break interop with some older servers that implement caldav-sched-04.

Comment 10

[Bruno Browning]

Attachment: Don't use RECEIVED-DTSTAMP or RECEIVED-SEQUENCE

Jdelic, are you in a position to build and test? I think this patch will likely solve the problem but I'm not in a position to test against Sun.

Comment 11

[Jdelic]

unfortunately I can't build, but I _can_ test. I also can just provide you with a test account on a server over here. I'll contact you via email with the login details.

Comment 12

[Peter Hulst]

I'm getting the same error when trying to accept a meeting update using Thunderbird 3.0.1 / Lightning 1.0b1. 

The error I see in the error console is similar to the one above: 

Error: An error occurred when writing to the calendar Zimbra! Error code: MODIFICATION_FAILED. Description: 
Source File: file:///Users/phulst/Library/Thunderbird/Profiles/6i96h0q5.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/modules/calUtils.jsm -> file:///Users/phulst/Library/Thunderbird/Profiles/6i96h0q5.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calCalendarManager.js
Line: 1000

Accepting new meetings and meeting cancellations works fine, it's just the modifications that don't work. 

We are on Zimbra though, not on Sun Calender Server. (unless Zimbra uses Sun's server, which I'm not sure about).

Comment 13

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Malintha, this seems to be one of the things changed in the scheduling rfc, can you take a look at this? See comment 9.

Comment 14

[Malintha Fernando [:marlio]]

HI Philipp, yes, it is something removed on the way to final RFC. I think we can safely remove it without breaking others. Here are more discussion about it http://www.ietf.org/mail-archive/web/calsify/current/msg00328.html

Comment 15

[[:MakeMyDay]]

Attachment: bug534228-debitrotted-V1.diff

Debitrotted patch. A try build is currently running: https://treeherder.mozilla.org/ui/#/jobs?repo=try-comm-central&revision=2b1420ef9634

Comment 16

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 8520905 [details] [diff] [review]
bug534228-debitrotted-V1.diff

Review of attachment 8520905 [details] [diff] [review]:
-----------------------------------------------------------------

r- for now pending the x-moz prop:

::: calendar/base/modules/calItipUtils.jsm
@@ +816,2 @@
>   *
> + * @param item either    calIItemBase

-either

@@ +819,4 @@
>   */
>  function setReceivedInfo(item, itipItemItem) {
>  
> +    item.setProperty("X-MOZ-RECEIVED-SEQUENCE",

What is it about the X-MOZ version of this prop. Why do we need it?

::: calendar/libical/design-data/params-in-prop.txt
@@ +1,3 @@
>  ACTION               VALUE X
>  ATTACH               FMTTYPE ENCODING VALUE X
> +ATTENDEE             CN CUTYPE DELEGATED-FROM DELEGATED-TO DIR LANGUAGE MEMBER PARTSTAT ROLE RSVP SENT-BY SCHEDULE-STATUS SCHEDULE-AGENT SCHEDULE-FORCE-SEND X

I'd suggest keeping it here, otherwise the ical parser will barf on the props if brought in from another source

Comment 17

[[:MakeMyDay]]

Attachment: bug534228-debitrotted-V2.diff

The original patch still contained a call to add those properties as attendee param, which now is removed.

The X-MOZ props can be removed in theory, allthough they have been in the code base before the attendee props were implemented. From what I see in the code, they are used to make sure that invitations are processed in the correct sequence, even if standardized properties for sequence or timestamp are not contained in an invitation, allthough I'm not sure if the ICS would be parsed at all in that case. If we decide to remove them, we should add a test here.

Regarding the removal of the params in the ical definition, I suggest to remove them, but we should make sure to make Libical to be more gracious with incorrect ics.

Ignoring inappropriate properties/parameters would be an asset instead of crashing TB. There are alss some other bugs arround, where malformed events are crashing the application, even though I have no bug number at hand atm.

Philipp, what do you think about this?

Comment 18

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

(In reply to MakeMyDay from comment #17)
> The X-MOZ props can be removed in theory, allthough they have been in the
> code base before the attendee props were implemented. ... If we decide to
> remove them, we should add a test here.
I would be in favor of getting rid of them.

> 
> Regarding the removal of the params in the ical definition, I suggest to
> remove them, but we should make sure to make Libical to be more gracious
> with incorrect ics.
This is a constant problem, yes. I've already set an option to treat unknown properties as IANA properties, which brought a lot of success, but I recall there was a problem with the ACKNOWLEDGED property in VALARM components not being detected by libical and therefore being removed, see bug 702206.


> Ignoring inappropriate properties/parameters would be an asset instead of
> crashing TB. There are alss some other bugs arround, where malformed events
> are crashing the application, even though I have no bug number at hand atm.
Maybe we should just add a unit test that tries a few combinations of unknown properties and makes sure they aren't removed during roundtripping. I think we might have an existing unit test we could add to.

Comment 19

[Karl Rossing]

Is there a lightning build available so that I can test the patch against Oracle CALDAV 7.0.4.16.0.

Our issue is with multiple X-Received-Sequence and not X-MOX-SEQUENCE