465853 - Events with UID that contains '/' are not displayed

Status: ASSIGNED

(Calendar :: Provider: CalDAV, defect)

Description

[Bernard Desruisseaux]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.18) Gecko/20081029 Firefox/2.0.0.18
Build Identifier: Lightning 0.9

While testing Lightning's CalDAV support with Google Calendar I noticed that a bunch of meetings weren't displayed in Lightning. Turns out the meetings that were not displayed had UID values that looks like:

UID:20070215-0030@example//[e-mail]

Most likely the '/' characters are causing issues here.

I can confirm that Google Calendar is actually returning those calendar components to Lightning and is properly percent encoding the '/' in their URL.

  <D:response>    <D:href>/calendar/dav/abcdef012345%40group.calendar.google.com/events/20070215-0030%40example%2F%2Fabcdef012345%40group.calendar.google.com.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <D:getetag>1-63327058174</D:getetag>
      </D:prop>
    </D:propstat>
  </D:response>

Reproducible: Always

Steps to Reproduce:
1. Create event with a UID value that contains a '/' in a CalDAV calendar collection.

Actual Results:  
Mozilla Lightning doesn't display the event.

Expected Results:  
Event should be displayed.

I've lost track of how these events were imported into the calendar.
I'm pretty sure the events originally had UID such as:

UID:20070215-0030@example

and that Google must have changed them automatically to

UID:20070215-0030@example//[e-mail]

It's possible that I actually moved those events from one calendar to another at some point and that would be when Google changes their UID... Don't know for sure. Anyway, only Google would know and care about this...

Comment 1

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

I'm not quite sure this is the reason, but I believe it may have to do with the way we separate the id from the rest of the url.

Needs investigation...

Comment 2

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Bernard, is it possible that you could invite me to the event with the strange id (kewisch.moz at gmail.com) ?

Otherwise, it would be nice if you could provide some more debug output so I can find out exactly where its failing...

Comment 3

[Bernard Desruisseaux]

Philipp,

I just added you as a "guest" to one of those events.

Comment 4

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Thanks Bernard.

Some debugging shows that while the event is shown during the getetag query, it is not found on the server when doing the calendar-multiget. The same happens on a simple get.

I've extended http://code.google.com/p/google-caldav-issues/issues/detail?id=31 to track the issue on the server side.

Comment 5

[Daniel Boelzle [:dbo]]

I think this needs to be tracked for 1.0, at least relnoted.

Comment 6

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

This is a Google Server issue against which we can't do a thing. I'm fine with relnoting though.

Comment 7

[Alois Schloegl]

I see also the phenomenon that VEVENT's with an UID containing a slash "/" are not displayed. However, I do not use any Google calendar, but the ical data is experted from phpScheduleIt and imported in DavICal server. 

Events with an UID like this are not displayed: 
UID:503ca6c43ab87&https://intranet.ist.local/scheduleit2/Web

After removing the slashes, the events are displayed. 
UID:503ca6c43ab87&https:==intranet.ist.local=scheduleit2=Web
UID:503ca6c43ab87&intranet.ist.local/scheduleit2/Web

I've also checked with RFC 2445 and 5545, and the forward slash ASCII(x2f) is part of the TSAFE-CHAR set. 

Therefore, I'm pretty sure this is not google server issue, but an issue of the mozilla calendar client. 

For testing, I've been using Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.16) Gecko/20120724 Iceowl/1.0b1 Icedove/3.0.11

Comment 8

[Alois Schloegl]

Another observation is, that this bug is only triggered when importing the calender using the format "CalDav"; when using the option "iCalendar (ICS)", there is no problem and all events are properly shown. 

Perhaps this can help fixing the problem.

Comment 9

[Alois Schloegl]

(In reply to Alois Schloegl from comment #7)

.. of course, this does also not work, 

> UID:503ca6c43ab87&intranet.ist.local/scheduleit2/Web

It should read like this:  
UID:503ca6c43ab87&intranet.ist.local

Comment 10

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Its been a while since I checked this, but I think it had to do with how Google responds to certain calendar queries that contain the UID.

Could you enable calendar.debug.log and calendar.debug.log.verbose in the advanced config editor, do an update (preferably on a new calendar that only has this one event, otherwise you will get a lot of logging) and post the log messages here? The ones that contain the UID should be enough.

Comment 11

[Alois Schloegl]

Attachment: messages in error console when loading event with and without slashes in UID from davical server

The attachment contains the log messages from the console. It contains two parts, the first part are the messages when a new calendar in davical was imported containing a single event with an UID containing slashes, namely:    

   UID:50697be34fee2&https://intranet.ist.local/scheduleit3/Web

The second part contains the log, when the same event but with 

   UID:50697be34fee2&intranet.ist.local

was important into davical, and then the calendar was reloaded. 
Only after the second step was performed, the event became visible in the client. 

Of course, the problem might be also on the server side (we are using davical  0.9.9.3 from debian squeeze). Either the mozilla client or davical does not handle forward slashes in a correct way.

Comment 12

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Alois, it looks to me like the event is not even being returned in the XML, this means it might be a server issue. I think this deserves a new bug, possibly with davical.

Comment 13

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Attachment: Always Encode Paths - v1

Not requesting review for this yet, since there is more testing to do. Specifically:

* On a Davmail Server, where the item path contains the title, does adding an event with an "ü" in its name still work (compare bug 575782).

* Does upgrading with a cached calendar that saves the location path decoded work, i.e is it still possible to: change events client side, change events serverside then update, ...

* Any other regressions with different caldav servers?

Comment 14

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Alois, can you give this patch a test run?

Comment 15

[Alois Schloegl]

I'm unable to comment on this patch. Sorry.

Comment 19

[BenjiM]

Is there some update on this bug?

This error also appears, if I try to import an event from xing to a (remote) calendar on a owncloud server, as the ics-file contains a UID with slashes.

Comment 20

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Nothing new currently. The concept of the patch still applies, though it likely doesn't apply to code any longer. The change in that patch could potentially be breaking so I'd rather not land it without some sort of testsuite for caldav. Thank you for the additional use case though!

Comment 21

[Florian Holeczek]

Really, what is this? This bug is 12 years old. I opened a bug on the same topic 4 years ago, which was closed as duplicate 3 years ago.

Every now and then, I have to patch ICS calendar files manually before importing an appointment. There are several business relevant sites out there which generate ICS files this bug applies to.

Folks... Core functionality of the product is not working, and, as it seems, the implementation is not standard-conformant either, since a slash seems to be a valid character.
From user perspective, I consider this a "fix on next release" topic. If I had to take a business decision whether to use Thunderbird as the company mail and calendar solution, this topic would be a showstopper for me.

I voted for this topic, but I don't think that the low number of votes reflects the importance. Few users will be able to track down their "it just won't import" experience to this bug entry.

Comment 22

[Magnus Melin [:mkmelin]]

Geoff added a CardDAV fake server. I suppose we could so something similar for CalDAV.
https://searchfox.org/comm-central/search?q=CardDAVServer.jsm&path=

What would it be testing? That saving an event works and viewing it later works with encoded path?

Comment 23

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Oh hey, Paul and I wrote a testsuite that I forgot about :D https://searchfox.org/comm-central/source/calendar/test/unit/test_caldav_requests.js

A bunch of tests around decoding and encoding would be good. the multiget and getetag requests can have encoded or decoded paths, and subsequent modifications should use the right path. It would make sense to emulate the responses of a specific server that shows this behavior.