1197320 - Free/Busy information of CalDAV calendar is no longer displayed after latest Lightning Calendar's update (v4.0.2)

Status: RESOLVED FIXED

(Calendar :: Provider: CalDAV, defect)

Description

[u547509]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0 Iceweasel/31.8.0
Build ID: 20150701023259

Steps to reproduce:

Client-side OS platform: MS Windows 7 SP1
Client-side software involved: Lightning 4.0.2 running on Mozilla Thunderbird 38.* (tested with 38.0.1, 38.1.0 and 38.2.0)

Server-side OS platform: Debian GNU/Linux 8 (Jessie)
Server-side software involved: DaviCal 1.1.3.1

Steps to reproduce the bug:

1. Install/Activate Lightning 4.0.2 Add-on on any supported Thunderbird version (38.*)
2. Add a CalDAV calendar if not already present
3. Create a new event
4. Click on "Invite atendees"
5. Select an atendee


Actual results:

Actual result: The atendee's free/busy information no longer appear


Expected results:

Expected result: The atendee's free/busy information is properly displayed

Important Remark: This bug was not present on previous Lightning versions (<= 4.0.1.2)

Comment 1

[[:MakeMyDay]]

Do you have any messages in the error console when reproducing (ctrl+shift+j)?

Do you have network access when trying this? Is offline support enabled or disabled? Or in short: are you affected by bug 1195974 when reproducing the issue?

Comment 2

[u547509]

There is one message on the error console that appears after clicking on "Invite atendees". My Thunderbird installation is not in english, but can be translated as:

<<The use of "Mutation Events" is deprecated. It is better to use "MutationObserver".
chrome://calendar/content/calendar-event-dialog-attendees.js		Line:  32 >>

However, I want to point out that this message also appears on the previous Lightning version (4.0.1.2), where the atendee's free/busy information is properly displayed. Thus it seems unrelated to the issue.

I have network access when trying this. The issue is present either if offline support is enabled or disabled.

On the Lightning versions not affected by this bug, when I'm typing the attendee's email address, I can see how the timetable where the free/busy information is shown "blinks". I guess that while typing, Lightning is querying the CalDAV server, trying to find a match for the typed email. When the whole email address has been entered and it is found on the CalDAV server, then the free/busy info of the atendee is shown. However, in 4.0.2 I'm not able to see this "blinking" behaviour anymore, it looks like Lightning is no longer querying the server while typing.


About the bug 1195974, yes I'm affected too. However the described behaviour is only present after a Thunderbird restart. I mean that, if Thunderbird is already open and I disable the network, my CalDAV calendar continue to be displayed, and I can even create new events. The calendar is displayed even after clicking on "synchronize calendars". However, if Thunderbird is restarted (while the network is already disabled), then my CalDAV calendar presents the behaviour described by the bug 1195974. A yellow triangle appears next to the calendar's name showing that "the calendar is temporarily unavailable", and no events are displayed. The events created after network disconnect and before TB restart are lost (I mean, they are not saved locally and written to the CalDAV server once the network is again available).

Comment 3

[u547509]

 -- correction about my last comment on bug 1195974: I have now tried to write an event while the network is down (but before Thunderbird's restart, thus when the calendar events are still displayed), and after reconnecting the network, the event appears. Thus the events created on offline mode aren't lost. --

Comment 4

[[:MakeMyDay]]

Thanks. No need to discuss the issue of bug 1195974 within this one. This is most probably a side effect of that bug. But to be sure there's no other issue, can you please enable calendar.debug.log and calendar.debug.log.verbose in the config editor and post what you get in the log when issuing a FB query? I was not able to reproduce the issue with a similar setup with an older Davical version.

Comment 5

[u547509]

After following your instructions the error console display far more information before clicking on "Invite atendees", but there is no single message logged after clicking on "Invite atendees" beside the one that I posted on my first answer ("The use of Mutation Events is deprecated"), even when the email address of the atendee is being typed or is already typed.

However, when I do the same process with 4.0.1.2, there are a lot of VFREEBUSY requests recorded as messages on the error console. The first one is sent to myself, because as a organizer of the event my email address appear by default on the top of the atendee's list (and my free/busy information immediately appear on screen). And then requests are sent to the email addresses being entered (and their free/busy info appears when they exist on the CalDAV server).

Thus I can only post the messages that appear on Lightning 4.0.1.2 after clicking on "Invite atendee" and do --not-- appear on Lightning 4.0.2 :



CalDAV: send (Originator=mailto:[e-mail],Recipient=mailto:[e-mail]): BEGIN:VCALENDAR
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
VERSION:2.0
METHOD:REQUEST
BEGIN:VFREEBUSY
DTSTAMP:20150823T095400Z
ORGANIZER:mailto:[e-mail]
DTSTART:20150821T220000Z
DTEND:20150906T220000Z
UID:e95df673-0ec2-4f86-87b5-706387eb8e9a
ATTENDEE;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT;CUTYPE=INDIVIDUAL:mail
 to:[e-mail]
END:VFREEBUSY
END:VCALENDAR

--------------------------------------------------------------------------------------------------
-------	NEXT MESSAGE RECORDED ON "error console" FOLLOWS	----------
--------------------------------------------------------------------------------------------------

CalDAV: recv: <?xml version="1.0" encoding="utf-8" ?>
<C:schedule-response xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav">
 <C:response>
  <C:recipient>
   <href>mailto:[e-mail]</href>
  </C:recipient>
  <C:request-status>2.0;Success</C:request-status>
  <C:calendar-data>BEGIN:VCALENDAR
PRODID:-//davical.org//NONSGML AWL Calendar//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REPLY
BEGIN:VFREEBUSY
DTSTAMP:20150823T095425Z
DTSTART:20150821T220000Z
DTEND:20150906T220000Z
FREEBUSY:20150822T120000Z/20150822T123000Z
FREEBUSY:20150822T143000Z/20150822T150000Z
FREEBUSY:20150824T091500Z/20150824T094500Z
FREEBUSY:20150827T061500Z/20150828T061500Z
FREEBUSY:20150828T103000Z/20150828T110000Z
FREEBUSY:20150902T064500Z/20150902T071500Z
UID:e95df673-0ec2-4f86-87b5-706387eb8e9a
ORGANIZER:mailto:[e-mail]
ATTENDEE;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT;CUTYPE=INDIVIDUAL:
 mailto:[e-mail]
END:VFREEBUSY
END:VCALENDAR
</C:calendar-data>
 </C:response>
</C:schedule-response>

--------------------------------------------------------------------------------------------------
-------	NEXT MESSAGE RECORDED ON "error console" FOLLOWS	----------
--------------------------------------------------------------------------------------------------

CalDAV: send (Originator=mailto:[e-mail],Recipient=mailto:[e-mail]): BEGIN:VCALENDAR
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
VERSION:2.0
METHOD:REQUEST
BEGIN:VFREEBUSY
DTSTAMP:20150823T095607Z
ORGANIZER:mailto:[e-mail]
DTSTART:20150821T220000Z
DTEND:20150906T220000Z
UID:938a0f96-e6e8-405a-a080-4af582ae3c48
ATTENDEE;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT;CUTYPE=INDIVIDUAL:mail
 to:[e-mail]
END:VFREEBUSY
END:VCALENDAR

--------------------------------------------------------------------------------------------------
-------	NEXT MESSAGE RECORDED ON "error console" FOLLOWS	----------
--------------------------------------------------------------------------------------------------

CalDAV: recv: <?xml version="1.0" encoding="utf-8" ?>
<C:schedule-response xmlns="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav">
 <C:response>
  <C:recipient>
   <href>mailto:[e-mail]</href>
  </C:recipient>
  <C:request-status>2.0;Success</C:request-status>
  <C:calendar-data>BEGIN:VCALENDAR
PRODID:-//davical.org//NONSGML AWL Calendar//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REPLY
BEGIN:VFREEBUSY
DTSTAMP:20150823T095632Z
DTSTART:20150821T220000Z
DTEND:20150906T220000Z
FREEBUSY:20150822T113000Z/20150822T120000Z
FREEBUSY:20150822T121500Z/20150822T124500Z
FREEBUSY:20150822T141500Z/20150822T144500Z
FREEBUSY:20150822T151500Z/20150822T154500Z
UID:938a0f96-e6e8-405a-a080-4af582ae3c48
ORGANIZER:mailto:[e-mail]
ATTENDEE;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT;CUTYPE=INDIVIDUAL:
 mailto:[e-mail]
END:VFREEBUSY
END:VCALENDAR
</C:calendar-data>
 </C:response>
</C:schedule-response>

Comment 6

[Sébastien Le Ray]

Same issue here with Davical 1.1.3.1-1 (debian package), TB 38.2.0, lightning 4.0.2.
No trace in log except the Observer warning, no server access

Comment 7

[Sébastien Le Ray]

After some dirty tracking by randomly adding logs, it seems that the issue is here http://hg.mozilla.org/comm-central/annotate/e6c5bce2b8fb/calendar/providers/caldav/calDavCalendar.js#l2402
getFreeBusy seems to assume that aCalId is a mailto: URI whereas it's not (no mailto prefix in my tests).

Comment 8

[Stefan Sitter [:ssitter]]

Might be regression from Bug 1048035. I already asked for comment in that bug.

Comment 9

[Sébastien Le Ray]

You're right, it's a regression from Bug 1048035, prependMailTo (http://hg.mozilla.org/comm-central/file/e6c5bce2b8fb/calendar/base/modules/calUtils.jsm#l296) is buggy:

(!aId.search(/^MAILTO:/i) && aId.includes("@")) ? "MAILTO:" + aId : aId;

String.prototype.search returns -1 not false when regexp doesn't match. Which is lucky since, String.prototype.includes does not exist… This should read

(aId.search(/^MAILTO:/i) == -1 && aId.indexOf("@") >= 0) ? "MAILTO:" + aId : aId;

Comment 10

[[:MakeMyDay]]

Attachment: FixFBLookup-V1.diff

Patch for cc, aurora and beta fixing the wrong check for search() result.

Comment 11

[[:MakeMyDay]]

Attachment: FixFBLookup-esr38-V1.diff

Fix for esr38, additionally using contains() instead of includes() (separate patch neccessary due to the switch from conatins to includes in version 40).

Comment 12

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 8653738 [details] [diff] [review]
FixFBLookup-V1.diff

Review of attachment 8653738 [details] [diff] [review]:
-----------------------------------------------------------------

::: calendar/base/modules/calUtils.jsm
@@ +293,5 @@
>       * @param  {string}        the string to prepend the prefix if not already there
>       * @return {string}        the string with prefix
>       */
>      prependMailTo: function(aId) {
> +        return (aId.search(/^MAILTO:/i) == -1 && aId.includes("@")) ? "MAILTO:" + aId : aId;

I have a bold suggestion, to be sure we should probably not use it on ESR:

return aId.replace(/^(?!MAILTO:)(.*)@/i, "MAILTO:$1@");

This is faster than the previous solutions on jsperf:
http://jsperf.com/regex-vs-lowercase-and-startswith

Hope I didn't miss anything :)

Comment 13

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 8653741 [details] [diff] [review]
FixFBLookup-esr38-V1.diff

Review of attachment 8653741 [details] [diff] [review]:
-----------------------------------------------------------------

r+ for the esr38 variant.

Comment 14

[[:MakeMyDay]]

Attachment: FixFBLookup-esr38-V2.diff

Updated patch for esr38: changed prepended mailto: to lowercase. Additionally, a fix for calItipUtils was added, where the email property of the msgIAddressObject must be checked to determine the invitation sender.

Comment 15

[[:MakeMyDay]]

Attachment: FixFBLookup-V2.diff

The same as for ers38, but with the above comment for using a regex considered, although the regex is slightly modified to catch all cases. Additionally, a test for removeMailTo and prependMailTo is added.

A try build for the test is currently running:

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=8fa1981cd14d

Comment 16

[[:MakeMyDay]]

Comment on attachment 8654636 [details] [diff] [review]
FixFBLookup-V2.diff

The test failed. I will provide an updated patch tomorrow once it passed. Removing the review request until then.

Comment 17

[Sébastien Le Ray]

Your remove test uses prepend instead of remove.
Moreover you could add a { input: "mailto:there.is.no.at", expected: "mailto:there.is.no.at" } (to cover stupid but handled case)

Comment 18

[[:MakeMyDay]]

Attachment: FixFBLookup-V3.diff

Updated patch with fixed test. I haven't picked up Sebastiens suggestion to add a bogus test, because such case would for the functions standalone but not in a prepend after remove scenario and would thereby break the overall use case anyway (beside that using a mailto prefix for a non email address would not be standard complient iirc).

Comment 20

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 8655001 [details] [diff] [review]
FixFBLookup-V3.diff

Review of attachment 8655001 [details] [diff] [review]:
-----------------------------------------------------------------

::: calendar/test/unit/test_calutils.js
@@ +6,5 @@
> +
> +function run_test() {
> +    removeMailTo_test();
> +    prependMailTo_test();
> +    run_next_test();

run_next_test not needed since the test is synchronous

@@ +9,5 @@
> +    prependMailTo_test();
> +    run_next_test();
> +}
> +
> +// tests for calutil.jsm

calUtils.jsm, but you might as well remove the comment.

@@ +19,5 @@
> +                {input: "first.last.example.net", expected: "first.last.example.net"}];
> +    for (let test of data) {
> +        equal(cal.removeMailTo(test.input), test.expected);
> +    }
> +};

No need to add semicolons here

@@ +29,5 @@
> +                {input: "first.last.example.net", expected: "first.last.example.net"}];
> +    for (let test of data) {
> +        equal(cal.prependMailTo(test.input), test.expected);
> +    }
> +};

No need for semicolons

Comment 21

[[:MakeMyDay]]

Attachment: FixFBLookup-V4.diff

Updated patch with comments considered.

Comment 22

[aleth [:aleth]]

https://hg.mozilla.org/comm-central/rev/f48fd117644f24c1f2d91a157ce31d3ee64c00d5
Bug 1197320 - Free/Busy information of CalDAV calendar is no longer displayed after latest Lightning Calendar's update (v4.0.2). r=philipp

Comment 23

[aleth [:aleth]]

Please set approval? flags on patches that need uplift.

Comment 24

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

adding checkin-needed for the backports

Comment 25

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Backported to releases/comm-aurora changeset 32968d0d7746

Comment 26

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Backported to releases/comm-beta changeset e7b0fc9b04af

Comment 27

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Backported to releases/comm-esr38 changeset 2ac04a2aa9de