Open Bug 306495 Opened 14 years ago Updated 4 months ago

autodetect remote calendar type so user doesn't need to pick (with DNS or .well-known)

Categories

(Calendar :: Provider: CalDAV, enhancement)

enhancement
Not set

Tracking

(Not tracked)

People

(Reporter: dmose, Assigned: pmorris)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Whiteboard: [no l10n impact])

Attachments

(5 files, 3 obsolete files)

When selecting a remote calendar, it's often not obvious to a user what WebDAV
and CalDAV mean, and which they should pick.  It should be possible to
autodetect whether a URL supports CalDAV or not.  We should do so, and make this
piece of UI go away.
Additionally it would be great if WebDAV servers are detected. That is, the
panel detects that the URL is not CalDAV but _is_ a generic WebDAV collection,
it should  bring up a WebDAV browser allowing the user to navigate to a CalDAV
(or GroupDAV ;-) collection.

Suggested sequence:
a) check for CalDAV using OPTIONS
   - if its a "leaf" CalDAV collection, directly subscribe
   - if its a collection containing CalDAV folders, show a panel with the 
     available calendars for subscription
b) check for WebDAV by trying to issue a PROPFIND (and handling a HTTP 501)
   - show a panel for traversing the WebDAV hierarchy, highlight CalDAV,
     GroupDAV collections and c)-like entities for subscription
c) try a GET and check whether the result is text/vcalendar or text/xcal or 
   whatever is understood
   - directly subscribe the GET/PUT-all-in-one calendar
c) show an error
Flags: blocking0.3?
Would accept a patch in the very near future for this.  Without fairly broad test-coverage to ensure we get this right, we won't take it though.
Flags: blocking0.3? → blocking0.3-
Cleaning up incorrectly assigned bugs; search for dmosecleanup to get rid of this bugmail.
Assignee: dmose → nobody
see bug 355270 comment 12 (point 3 and 4) and following for a related discussion.
Proposing a slightly-revamped calendar creation UI to accommodate calendar-type detection. 

This has discovery of existing-but-unregistered CalDAV calendars, which I think we need to do, partly because Apple has made doing so standard. We could do something similar for ICS-on-WebDAV without changing this UI substantially (though we wouldn't be using a DAV:displayname property).
The proposal looks good for me. 

Some comments:
I would suggest to use check boxes instead of option buttons for "Choose an existing calendar". This would allow users to subscribe to more than in one single step.

It should also be possible to search/filter calendars on a specific server.

[ John                           ] [Search]

+--------------------------------+
| [ ] John Doe                   |
| [ ] John's Sports Calendar     |
|                                |
|                                |
|                                |
+--------------------------------+
While this proposal looks fine on first sight, it fails for non-url based providers. For example, given that I would want to create a wizard that lets users select their google calendars by only entering username and password, where would I hook that in?
(In reply to comment #6)

> I would suggest to use check boxes instead of option buttons for "Choose an
> existing calendar". This would allow users to subscribe to more than in one
> single step.
> 

I assume that we would then want a multiple-calendar Customize page, allowing the user to set display name / color / alarms for all the newly-subscribed calendars all at once? Not terribly simple, but I think preferable to a sequence of single-calendar Customize pages.

> It should also be possible to search/filter calendars on a specific server.
> 
> [ John                           ] [Search]
> 
> +--------------------------------+
> | [ ] John Doe                   |
> | [ ] John's Sports Calendar     |
> |                                |
> |                                |
> |                                |
> +--------------------------------+
> 

Searching is going to raise some new UI issues here: we won't be able to simply use the DAV:displayname property when displaying search results and when pre-filling the Name field on the Customize page, due to the inevitability of collisions in that space cross-principal (for instance, it's quite likely that several people will name their main calendar collection "Work Calendar"). So I think we'll want to in some fashion include an indication of the associated principal as well as the displayname. The obvious way of doing this would be:
(Christian Jansen) Work Calendar
Which is simple but expensive in terms of horizontal screen real estate. That's probably acceptable when displaying search results but probably not for use in the calendar list - and if we're going to pre-fill that Name field in the wizard we ought to do so with something usable. Another possibility would be to introduce some hierarchy into the calendar list, e.g.
Christian Jansen
|-Work Calendar
This burns screen space vertically instead and provides a familiar UI (similar to the Thunderbird accounts/folders pane). It would also allow us - if we decided we wanted to - to display the CalDAV Inbox/Outbox/Notifications folders in the calendar list. That latter is of questionable utility though. There's been some discussion of this in bug 407840. Also, I think that currently CalDAV is the only provider that would benefit from such a hierarchy.

(In reply to comment #7)
> it fails for non-url based providers

Good point. Perhaps that first page should read
0 On My Computer
0 At Google
0 Elsewhere on the Network

? I assume we'd want some way for any new non-url-based providers to plug themselves in here as well.
Blocks: 349793
I'm nominating this for the tb-integration flag. Our current calendar-creation process is a huge usability issue; we need to do better before Lightning lands in tb. I think we have a good idea what needs to be done here, at least initially, so it should not be too much of a stretch.
Flags: tb-integration?
Flags: tb-integration? → tb-integration+
Assignee: nobody → browning
Regarding the UI, I think we should reduce wizard steps if the wizard screeen space permits. Also, I think we might still want to allow the user to select the provider, but instead either preselect the right type after detection or provide an "advanced..." button to let the user change that selection.

Also, when fixing this bug, we should keep in mind bug 378873 and its dependant bug. If at all possible, we should allow providers to plug in somewhere in case they are not url dependant or need custom steps. This might partially be covered in the proposal from comment 8, but we need some ui-review there.

Regarding the possibility to create a new calendar or choose existing calendars, we may want to make this more pluggable as well. I can imagine that other server types can provide facilities to choose existing or create new calendars and maybe want to display more information (maybe a color, or a timezone) when choosing the calendar. In an early stage of the gdata provider I mocked up a wizard page that used a <tree> that contained the existing calendars' display name, color and timezone. Maybe something like this would be beneficial, allowing the provider to implement its own tree view to display the calendars, maybe providing base prototypes to ease things for beginners.
I think the next step for implementation is to think of some sort of API that allows extensions and shipped providers to determine the type of provider from the uri, with some sort of importance level. Imagine the following scenario:

https://www.google.com/calendar/dav/example@example.com/user/
* The caldav provider will try to detect this and fail due to a Google bug.
* A separate implementation of the API could detect that this is specifically a
  Google CalDAV server and return "caldav".

For the importance I currently don't have a specific use case, but in general I can imagine:

* Provider A detects a certain URL to be itself and returns "providerA"
* Provider B does a much better job since it has server-specific extensions to
  whatever Provider A does and returns "providerB".

Without importance, depending on in which order providers are enumerated, providerA may win even though providerB does a better job.

Querying the provider should be done asynchronously, since its possible that the detection mechanism needs to do further HTTP requests to find out if its the correct provider.
Duplicate of this bug: 446099
From bug 446099, this autodetection code should also handle HTTP errors like 404, telling the user that no calendar resources could be found at the location provided.
Flags: wanted-calendar1.0+
It should be noted that multiple CalDAV server implementations (e.g., Apple, Google, Oracle, etc.) also provide support for Internet Calendar Subscriptions by allowing GET requests to be targeted at CalDAV calendar collections.

While the default should be to access the calendar collection with CalDAV when possible, users should have a way to specify that they simply want to subscribe to the calendar.

With Apple iCal this is a non issue since you are forced to specify a principal URL when configuring a CalDAV account.
I'm marking P1 based on the theory that this is pretty important for adoption - feel free to correct me if I'm wrong.
Priority: -- → P1
I think this is actually more than one bug. One part of it would be the caldav part of discovering the right calendar urls using OPTIONS, the other would be to revise the new calendar dialog to allow auto-config similar to what is anticipated for thunderbird.

In any case, I don't believe this is something that is required for Lightning being integrated into Thunderbird. I do believe it is beneficial and the caldav part should be blocking-calendar1.0+, but I think we can live with the old dialog for 1.0/Thunderbird3, if time doesn't permit.

Updating the new calendar dialog is something we'll need the thunderbird autoconfig dialog for. We should split this bug, I don't have the time to do it now though.
Assignee: browning → nobody
Whiteboard: [needs thunderbird autoconfig dialog]
Agreed that this might be more than one bug, and that you can get away with releasing 1.0 without it, but I think P1 is appropriate given the immediate pain that resolving it would relieve.

I agree with Bruno's comment above - Apple has made CalDAV discovery using the /principals/users/{username} URL the gold standard that everyone wants to follow.  I set up an iPhone with Zimbra CalDAV yesterday and it was both painless and pleasant.  The same cannot be said about calendar subscriptions in Lightning (1.0b2pre).
While it is true that Apple's search for /principals/users/{username} is first kid on the block, any implementation should consider that there is a standard under development that will replace it.

See the "Defining Well-Known URIs" Internet-Draft:

 http://tools.ietf.org/html/draft-nottingham-site-meta-04


Combine that with the current-user-principal extension (RFC5397) and there is a very straightforward path to finding a user's calendars once you have a server hostname and port to connect to.  Sprinkle an appropriate _srv lookup onto that and in due course it should be possible to find a user's calendars simply from the domain name and authentication details.
Lenni will be working on this for the 2011 Google Summer of Code. I'm looking forward to his work!
Assignee: nobody → Lennart.Bublies
Status: NEW → ASSIGNED
(In reply to Philipp Kewisch [:Fallen] from comment #20)
> Lenni will be working on this for the 2011 Google Summer of Code. I'm
> looking forward to his work!

2011 GSoC is over, did it actually happen?
Dave the result of the GSoC can be found as an attachment to bug #378873 as far as I know.
Dave, the result is indeed the extension mentioned there. I am still in contact with the student and he will be continuing his work soon. I have found the extension to work fine for some calendar servers, but it still needs refining for servers that do special things (i.e for Google servers)
Duplicate of this bug: 737424
Blocks: 883082
Firefox OS appears to implement CalDAV auto-provisioning (that is, you enter one URL, specify your credentials, and all your calendars are loaded).

Any chance the same back-end code, at least, can be used by Lightning? Who did that work?
Duplicate of this bug: 880099
Since Lightning's never going to get the cycles Mozilla will put behind Gaia, why not leverage their work for this? I can't believe the underpinning's of Gaia's calendar client are so different as to make that impractical.
Its not too much about the actual code to do the detection, but rather the code to hook it up to Lightning's quite historical architecture. Aside from what has been done in this bug, there needs to be some changes to make autodetection possible for all types of providers, then its trivial to put the existing code into the caldav provider.

In an ideal world we would also change the calendar manager to have accounts that have calendars, instead of a simple list of calendars.

And foremost, its a matter of finding someone with enough time to take care :)
Thanks, Philipp, that makes sense. It sounds like there could be some discussion (not in this bug), then, about the idea of perhaps rebasing Lightning off the calendar work in Gaia, to whatever extent possible.
Indeed, there is some code that could likely be used there. To get there we need to do some more backend changes though, b2g has the advantage that they got to reboot the whole architecture. First and foremost, we want to switch from libical to ical.js. It is already part of the production code, but there are still some blocking issues keeping us from enabling it by default.

I already have a work in progress caldav provider locally that uses the b2g-caldav code. Until we have ical.js enabled, it still is a bit cludgy (convert iCalendar from caldav -> jCal in b2g-caldav -> iCalendar to be usable with Lightning -> Lightning's objects). Also, some of the more advanced caldav features are not yet adapted.

There are surely some other areas we can and should make use of the b2g code, but again, we need developer to move in that direction. If this is something you'd like to help out with, please email me. Its a step by step process :)

I suggest further discussion either via email or in the mozilla.dev.apps.calendar newsgroup.
Priority: P1 → --
Duplicate of this bug: 883079
Duplicate of this bug: 883082
Duplicate of this bug: 861622
When finishing up this bug, the following things should be taken care of:

* caldav autodetection via well-known uris
* DNS SRV/TXT autodetection
* Trying common patterns

If any of this is not taken care of, followup bugs should be filed.
Keywords: student-project
Whiteboard: [needs thunderbird autoconfig dialog]
Summary: autodetect remote calendar type so user doesn't need to pick → autodetect remote calendar type so user doesn't need to pick (with DNS or .well-known)
Duplicate of this bug: 965686
Duplicate of this bug: 965686
Duplicate of this bug: 1183035
Duplicate of this bug: 1259300
Duplicate of this bug: 1294028
Duplicate of this bug: 1361035
I understand that commenting here when I don't contribute any code myself is not necessarily productive, but I just wanted to flag, that all CardDAV/CalDAV clients which appeared since this bug was first reported have auto discovery, neither on an iPhone, gnome-online-accounts, an iBook, DAVdroid, the CardBook addon for TB, not even Windows 10 do I have to provide the full URL to the individual calendars. So Lightning seems have vastly fallen behind in those 12 years. So how Caldav is handled in Lightning really feels so antiquated. As I am trying to get my organisation to use  Nextcloud  for joint scheduling, I am confronted daily with the fact that you only need to enter three things on most clients (URL, user name, password) to get all of your passwords and address books onto your device, but TB/Lightning is the negative exception to the rule. 

Since the CardBook extension already implements DAV discovery for TB, it can't be rocket science after all to adapt this code for Lightning. Maybe I'm wrong about that. But I have difficulties understanding why what is not a problem for CardBook should be so much more difficult for Lightning.
It is not rocket science, and I have some work in progress code that I never manage to finish up :-/ Sorry for the delays!
Philipp, what's the scope and state of your wip patch for this bug? For bug 1299610, I would need to fire an OPTIONS request to detect auto-scheduling capability of the respective calendar for a consistent UX across calendar setup and property editing.

If this bug is not only about detecting the appropriate base URI for a calendar principal for a calendar server, but also about providing a list of available calendars in a users collection within the wizard, such a detection is probably already part of your patch?
Flags: needinfo?(philipp)
(In reply to Philipp Kewisch [:Fallen] from comment #42)
> It is not rocket science, and I have some work in progress code that I never
> manage to finish up :-/ Sorry for the delays!

Hi Philipp, it was great to hear that someone is working on it at last. Did you by chance have the time to make any progress on that since you commented 5 months ago? Really sorry for nagging, but this would be one of the major issues with lightning being finally solved....
I'll upload my WiP patches soon. IIRC it was just a matter of polish and some cleanup, possibly some migration code. The code is complete from backend to UI. I'm not sure if it will allow for the use case in comment 43, but at least some of the calendar info will be available earlier on.

That said, I haven't touched them in 5 months. It would be nice to get this in, but of course it is too late for strings again. Maybe this will get easier with the new gecko-strings repo and we can ship it earlier.
I take back the strings part, I was thinking 57. We still have time until 59 hits beta.
This needs to be done in a separate patch, otherwise hg freaks out.
Attached patch Part 2 - Autodetection WiP - v1 (obsolete) β€” β€” Splinter Review
This is the main patch. It looks like I haven't taken care of l10n yet. The strings required are hardcoded. I see I used todo-l10n in some cases, but I'm not sure if this is all.

Given it is not a lot of strings we could probably pull these out to commit now, but I won't get to that before the end of the year. Anyone else interested?
Assignee: Lennart.Bublies → philipp
Flags: needinfo?(philipp)
Attached patch Strings - v1 (obsolete) β€” β€” Splinter Review
Here are the strings I'd like to push. The patch only contains additions to avoid running into trouble if I don't mak eit. If you would like to see them in action please check out this clickable mockup: https://app.moqups.com/kewisch/h5tL4vWrox/view/page/adcac4348
Attachment #8958308 - Flags: review?(makemyday)
Comment on attachment 8958308 [details] [diff] [review]
Strings - v1

Review of attachment 8958308 [details] [diff] [review]:
-----------------------------------------------------------------

r+, just some nits for consistency.

::: calendar/locales/en-US/chrome/calendar/calendarCreation.dtd
@@ +13,2 @@
>  
> +<!ENTITY locationpage.description           "Provide info about what is needed to access your remote calendar" >

Even if it has been that way before, maybe information is better then info here.

@@ +18,5 @@
> +<!ENTITY locationpage.location.placeholder  "URL or host name of the remote calendar service">
> +<!ENTITY locationpage.optional.placeholder  "optional"
> +
> +<!-- Discovery states -->
> +<!ENTITY discovery.inprogress.label "Please wait while your calendars are being discovered">

inprogress hasn't a closing dot other then the other discovery labels.

@@ +24,5 @@
> +<!ENTITY discovery.authfail.label "The username and password you have entered were not accepted. Please check your settings."
> +
> +<!-- Calendar selection -->
> +<!ENTITY selectcalendar.description "Please select the calendars you would like to subscribe to.">
> +<!ENTITY providertype.label "Calendar Type">

The other labels for input controls have a trailing :
Attachment #8958308 - Flags: review?(makemyday) → review+
Attached patch [checked in] Strings - v2 β€” β€” Splinter Review
Thanks for the review. I spent some time actually implementing the dialog to see how it feels and got some feedback from #maildev. I'm also opting for better string entity names instead of name consistency that is no longer correct.

I'm going to push strings v2 now.
Attachment #8958308 - Attachment is obsolete: true
Attachment #8958762 - Flags: review+
Pushed by mozilla@kewis.ch:
https://hg.mozilla.org/comm-central/rev/0c8cb75798fd
Add strings for calendar autodiscovery. r=MakeMyDay
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Obviously not fixed yet. The strings were technically pushed with late-l10n, so I am marking as such. Bug staying open for the actual patch which is 90% done. Need to handle some of the edge cases.
Status: RESOLVED → REOPENED
Keywords: late-l10n
Resolution: FIXED → ---
Attachment #8958762 - Attachment description: Strings - v2 → [checked in] Strings - v2
Attachment #8958762 - Flags: approval-calendar-beta+
Attachment #8936131 - Attachment is obsolete: true
Attachment #8936130 - Attachment is obsolete: true
Status: REOPENED → ASSIGNED
You were talking about porting in this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=467153#c15

Not sure if that has happened here yet or not.
Duplicate of this bug: 1496744
Assignee: philipp → nobody
Status: ASSIGNED → NEW
Type: defect → enhancement

This patch depends on bug 1546606 and applies on an older version of c-c, please see bug 1546606 for the parent. I'm dumping it here because it doesn't look like I will get to this any time soon, but I'd like it to be finished. Magnus, can you find someone to work on this?

There are two parts to it, mainly because hg got confused when I had them both in one. Maybe the hg mv'ing around is not really great. I'm fine adapting the filenames differently if necessary.

Assignee: nobody → philipp
Flags: needinfo?(mkmelin+mozilla)

This is the actual meat. What we may want to do is de-xpcom the new classes, as we're aiming to get rid of it. I'm fairly certain there are no tests in this patch, but some of those would be good too.

-> Paul.

Assignee: philipp → paul
Flags: needinfo?(mkmelin+mozilla)
Whiteboard: [no l10n impact]
You need to log in before you can comment on or make changes to this bug.