Closed Bug 920755 Opened 11 years ago Closed 11 years ago

Google CalDAV OAuth2 calendars fail to load [Error doing webdav sync: 400]

Categories

(Calendar :: Provider: CalDAV, defect)

Lightning 2.6
defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bmurali, Assigned: Fallen)

References

Details

(Whiteboard: [wanted-2.6.x])

Attachments

(1 file, 2 obsolete files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Firefox/24.0 (Beta/Release)
Build ID: 20130910160258

Steps to reproduce:

I am subscribed to two Google calendars using CalDav (NOTE: I do not use the Provider addon)


Actual results:

Every time I start Thunderbird (24 beta), it gives me a popup asking if I should give permission to access these Google calendars.  I give the access, but most of the times it gives a yellow error indication next to each calendar and does not load their contents properly.  In particular it shows the following on the error console:


Error: 'Component is not available' when calling method: [nsIActivityManager::removeActivity] = NS_ERROR_NOT_AVAILABLE
Source file: resource://app/modules/activity/autosync.js
Line: 203


Error: [calCachedCalendar] replay action failed: null, uri=https://apidata.googleusercontent.com/caldav/v2/bmurali5128@gmail.com/events, result=2147500037, op=[xpconnect wrapped calIOperation]


This is the same whether I choose the offline mode or not


Expected results:

The following:

 1. The calendars should load properly and sync up after I have given it permission
 2. Ideally, it should only ask for permission once and subsequent times it should just use the stored credentials in the password manager
 3. The password manager keeps adding a new credential entry every time I start Thunderbird and click "Accept" in the permissions pop up from Lightning
It appears that Google CalDAV now supports webdav-sync, and loading fails with:

CalDAV: webdav-sync Token: null

followed by:

Warning: CalDAV: Error doing webdav sync: 400
Status: UNCONFIRMED → NEW
Ever confirmed: true
Assignee: nobody → matthew.mecca
Severity: normal → major
Summary: Google Caldav calendars fail to load properly → Google CalDAV calendars fail to load [Error doing webdav sync: 400]
Attached patch Fix v1 (obsolete) β€” β€” Splinter Review
It looks like the sync-level element is required in the synchronization request, but we currently don't include it.

Per RFC6578 - http://tools.ietf.org/html/rfc6578#section-3.3

   Servers MUST support only Depth:0 behavior with the
   DAV:sync-collection report, i.e., the report targets only the
   collection being synchronized in a single request.  However, clients
   do need to "scope" the synchronization to different levels within
   that collection -- specifically, immediate children (level "1") and
   all children at any depth (level "infinite").  To specify which level
   to use, clients MUST include a DAV:sync-level XML element in the
   request.
Attachment #810335 - Flags: review?(philipp)
Attachment #810335 - Flags: approval-calendar-release?(philipp)
Attachment #810335 - Flags: approval-calendar-beta?(philipp)
Attachment #810335 - Flags: approval-calendar-aurora?(philipp)
Status: NEW → ASSIGNED
Yes, they just enabled this shortly, its very brand new.

I'll take a look at this for 2.6.1. Back when webdav-sync was implemented, sync-level did not exist. Could you check the other differences between the older drafts and the actual rfc to make sure we support everything?
Whiteboard: [wanted-2.6.x]
Matt, could you send me the exact request that does this? (URL, request headers, request body, response headers, response) I'll see if I can have a workaround put into place even before the release.
Might not need that, just saw the logs in the duplicate bugs. Will get back to you!
Have the same problem with two google calendars with Thunderbird 24 and Lightning 2.6 (Win7 64bit).
Google changed the dav: https://blog.mozilla.org/calendar/2013/09/google-is-changing-the-location-url-of-their-caldav-calendars/
 
So I changed to the new Google DAV and google wanted me to login and accept the access of lighting to my calendars. All worked fine

But now lightning can't access the calendar.

Errors (sry for german)

Zeitstempel: 26.09.2013 15:23:05
Warnung: getAttributeNodeNS() sollte nicht mehr verwendet werden. Verwenden Sie stattdessen getAttributeNS().
Quelldatei: resource://calendar/modules/calXMLUtils.jsm
Zeile: 32

[JavaScript Warning: "CalDAV: Error doing webdav sync: 400"]

[JavaScript Warning: "Fehler beim Lesen von Daten fΓΌr Kalender: Marc. Allerdings ist dieser Fehler wahrscheinlich vernachlΓ€ssigbar, daher versucht das Programm fortzufahren. Fehlercode: DAV_REPORT_ERROR. Beschreibung: Es wurde ein Fehler beim Lesen des Kalenders festgestellt: https://apidata.googleusercontent.com/caldav/v2/xxxxx@googlemail.com/events. Der Kalender wurde deaktiviert bis er sicher genutzt werden kann."]

[JavaScript Warning: "Fehler beim Lesen von Daten fΓΌr Kalender: Marc. Allerdings ist dieser Fehler wahrscheinlich vernachlΓ€ssigbar, daher versucht das Programm fortzufahren. Fehlercode: READ_FAILED. Beschreibung: "]

If u need translation, ask me. 
Short: Error by reading the calendar.
Summary: Google CalDAV calendars fail to load [Error doing webdav sync: 400] → Google CalDAV OAuth2 calendars fail to load [Error doing webdav sync: 400]
Blocks: ltn261
Matthew, fixing all the issues around CalDAV synch would be a major, major win given it's been buzzing in the web off and on since 2008 and there are SO many disparate bug reports.

I have at least one or two bugs in captivity right now, and am glad to help you get to the bottom of this.  I'm a techie but don't know Thunderbird internals, so would need steps, but can help gather and / or pick through logs, or whatever else you need.  I want to help you completely eradicate this issue.
The modification proposed on "Fix v1" has solved the problem for me on TB v24 + Lightning v2.6 Windows. Shared Google Apps calendars begin to be usable again.

Usually the calendars do not get synced at startup. A manual synchro seems to solve the problem.

Calendars where the user only shares availability and not detailed information are not always shynchronised (even with a manual synchronisation).
Fix v1 from matthew works fine for me. Thx a lot! 
Souldn't be a problem to include this single line in the program code.
I see the same problem as Marc above (this time in English):

[JavaScript Warning: "CalDAV: Error doing webdav sync: 400"]
[JavaScript Warning: "There has been an error reading data for calendar: Peter.  However, this error is believed to be minor, so the program will attempt to continue. Error code: DAV_REPORT_ERROR. Description: There has been an error reading data for calendar: https://apidata.googleusercontent.com/caldav/v2/<calid>/events. It has been disabled until it is safe to use it."]
[JavaScript Warning: "There has been an error reading data for calendar: Peter.  However, this error is believed to be minor, so the program will attempt to continue. Error code: READ_FAILED. Description: "]


I don't see any mentioning anywhere of 

CalDAV: webdav-sync Token: null

so I wouldn't know if this is the exact problem as described in this report. Comments, please ?


A little side-note for the Lightning developers:

When Googling for the problem I came across this Mozilla blog:
http://blog.mozilla.org/calendar/2013/09/google-is-changing-the-location-url-of-their-caldav-calendars/

Beware that this Blog will instruct you to use the following URL:

https://apidata.googleusercontent.com/caldav/v2/your-email-address/events

Technically this is not correct. What needs to be the second last element of that URL is not your e-mail address; it is your Google Calendar ID. This just HAPPENS in the typical case to be the same as your e-mail address but if you have multiple calendars on the same Google account then the two are not the same. Took me some time to figure that one out. You might want to change the text in the blog.

A Google Calendar ID is the same as your e-mail address for the Google account's primary calendar. For all other calendars on that account they look something like this:

 b2v45111o9ao1p39ctuh5ugk4x@group.calendar.google.com

Never mind, that is not the issue here. 

I believe some of the issues that you read about with regards to Lightning and Google CalDAV are related to people having a more complex setup than anticipated by the Lightning developers. Here are some pointers:

- Users will have multiple Google Calendar accounts and they will try to bring them all into Lightning. This is quite normal, I think. I remember the days where this would not work with Password Manager as you could store only one credential per site until you found that secret config setting somewhere. I wonder what the status is on that one and if the current change affects such scenario. (in fact I don't really understand what Lightning stores for me in Password Manager in this new OAuth scenario, but perhaps I'm not supposed to?  It looks like nonsense to me)

- Users will sometimes have multiple calendars on the same Google Calendar account. I believe this is probably less rare than having multiple Google Calendar accounts but nevertheless such scenario also need to be tested.

Hope this can help.
i tried it too:

BUT !!!

google calendar  is working after patch with CALDAV again (exclusiv appointment invites

BUT !
the other CALDAV servers are not working any more :( (eg: egroupware)

seems to be, that it must be added this extra parameter to check if it is a google CALDAV server, and only use this extra parameter ('<D:sync-level>1</D:sync-level>' +) if it is google

additionally as explained above, adding an appointment via email invite not working
if i click in the invitment on "accept" it ask me which calendar, then then i select the google CALDAV calendar, and then it comes:

Zeitstempel: 27.09.2013 15:16:46
Fehler: Beim Schreiben in den Kalender I-New CALDAV ist ein Fehler aufgetreten! Fehlercode: MODIFICATION_FAILED. Beschreibung: Status-Code: 2147500037, Die Anfrage kann nicht verarbeitet werden.

Status-Code: 2147500037, Die Anfrage kann nicht verarbeitet werden.

Server Replied with 404

in console logging is then:

Server Replied with 404
Quelldatei: resource://calendar/modules/calUtils.jsm -> file:///C:/Users/Camel/AppData/Roaming/Thunderbird/Profiles/9dvislam.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calCalendarManager.js
Line: 961 

=
[code]
        // Log warnings in error console.
        // Report serious errors in both error console and in prompt window.
        var isSerious = (aErrNo == calIErrors.MODIFICATION_FAILED);
        if (!isSerious) {
            WARN(summary);
        } else {
            // Write error to console.
            Components.utils.reportError(summary);
[/code]
mhh sometimes it still doesn't work, even with Fix v1 from Matthew. :-/ You need to restart then. Hope they'll fix this, soon.
There are a number of different issues here:

* Google introduced caldav v2, we adapted with the initial patch
* There is a bug on our side when the authentication token needs to be refreshed (bug 920230)
* Google afterwards introduced webdav-sync and doesn't react to the fallback of using the Depth
  header which we send (this bug)

Thanks for your input, especially on the egroupware support. I have everything else covered and will take care as soon as I can. If you can test this patch with other servers that would be nice!
Thanks Philipp. Your efforts are appreciated. I'll be happy to test out the scenarios I mentioned against Google Calendar as soon as there's a full patch release available.
(In reply to Peter from comment #11)

> - Users will sometimes have multiple calendars on the same Google Calendar
> account. I believe this is probably less rare than having multiple Google
> Calendar accounts but nevertheless such scenario also need to be tested.
> 

I am not sure. For me and some people I know, we use multiple Google Calendars with our own (one) Google account.
For instance: One calendar for Private/Personal use, one for Business 1, one for Business 2, and some others.
(I personally have 7 calendars on my Google account.)

And some do have additional another, external (shared) calendar included as well.

Just to let you know about quite common use-cases.

PS: Thanks for your specific help on this issue.
> Thanks for your input, especially on the egroupware support. I have
> everything else covered and will take care as soon as I can. If you can test
> this patch with other servers that would be nice!

I am also using two different calendar servers with Lightning, Google and the one from university. After the patch, Google works, but not the university calendar, which had worked before.

I'm also ready to check any new patches.
Comment on attachment 810335 [details] [diff] [review]
Fix v1

Aha, i see why this is not working. You use <D:sync-level>, but as we don't use a namespace prefix here and the default namespace is DAV:, it should be just <sync-level>1</sync-level>.

Could you folks testing this patch make the changes and see if it works with non-Google servers?
Attachment #810335 - Flags: review?(philipp)
Attachment #810335 - Flags: review-
Attachment #810335 - Flags: approval-calendar-release?(philipp)
Attachment #810335 - Flags: approval-calendar-beta?(philipp)
Attachment #810335 - Flags: approval-calendar-aurora?(philipp)
(In reply to Philipp Kewisch [:Fallen] from comment #18)
> Comment on attachment 810335 [details] [diff] [review]
> Fix v1
> 
> Aha, i see why this is not working. You use <D:sync-level>, but as we don't
> use a namespace prefix here and the default namespace is DAV:, it should be
> just <sync-level>1</sync-level>.
> 
> Could you folks testing this patch make the changes and see if it works with
> non-Google servers?

Hi Philipp,

I just applied your changes to fix v1. It still works with Google CalDAV v2. Regular CalDAV with an OwnCloud-Server functions as well.

In detail we tested synchronization with both servers and creating a new event. We also made sure, that is still works after a restart of Thunderbird.

Thanks for your efforts.
(In reply to Philipp Kewisch [:Fallen] from comment #18)
> it should be just <sync-level>1</sync-level>.
> 
> Could you folks testing this patch make the changes and see if it works with
> non-Google servers?

It works fine with both Google and the server from university.
Being an average TB-Lightning Google-Caldav user I have no idea whether / How I can implement this fix. 

Is this only possible for developers or also for avergae users?
The fix works for me on my several google calendars.
(In reply to JL from comment #21)
> Being an average TB-Lightning Google-Caldav user I have no idea whether /
> How I can implement this fix. 
> 
> Is this only possible for developers or also for avergae users?

On Linux the extension is per-user, so run:

find ~ -name calDavRequestHandlers.js

And edit the file to add the relevant line (see also Comment 18 above). Afterwards, restart Thunderbird.

Good Luck!
(In reply to JL from comment #21)
> Being an average TB-Lightning Google-Caldav user I have no idea whether /
> How I can implement this fix. 
> 
> Is this only possible for developers or also for avergae users?

The fix will of course also be released for non-developers in 2.6.1. I'm just trying to figure out what the right fix is before releasing a new version. Thanks for your patience!
****, i don'T understand, but now it is not working for.
Neither googe or egroupware....
but it msut be my problem, because others reported, that google is working with that changes .... strange ...

I'm blind ?

my code: in: calDavRequestHandlers.js
-------------------------------------

        let queryXml =
          xmlHeader +
          '<sync-collection xmlns="DAV:">' +
            syncTokenString +
              '<sync-level>1</sync-level>' +
              '<getcontenttype/>' +
              '<getetag/>' +
            '</prop>' +
          '</sync-collection>';

i can'T believe, that i made a mistake.

Also tried to remove the calendars, then restart TB, then re-add the calendars again .... strange ....
the only thing, that i'm also using the gdata calendar extension, too, but don'T think, that this is the issue ...
:(
strange ....

also found:
Zeitstempel: 30.09.2013 18:02:54
Warnung: Mit document.write() wurde ein nicht balancierter Baum geschrieben, was dazu gefΓΌhrt hat, dass Daten aus dem Netzwerk neu geparst werden mussten. FΓΌr weitere Informationen https://developer.mozilla.org/en/Optimizing_Your_Pages_for_Speculative_Parsing
Quelldatei: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=720563984538.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcalendar

any ideas - maybe i mad a mistake like a newby ?

@Phillipp: if wanted i can give you an account on my EGW server - if interessted, you can contact me on my email: camroATgmx.at

cu
Zeile: 62
> my code: in: calDavRequestHandlers.js
> -------------------------------------
> 
>         let queryXml =
>           xmlHeader +
>           '<sync-collection xmlns="DAV:">' +
>             syncTokenString +
>               '<sync-level>1</sync-level>' +
>               '<getcontenttype/>' +
>               '<getetag/>' +
>             '</prop>' +
>           '</sync-collection>';
> 
> i can'T believe, that i made a mistake.

It seems as if you added the line '<sync-level>1</sync-level>'  at the wrong place. Make sure that you add it precisely at the place that the patch tells. There are several occurences of "let queryXml".
thx, i have no idea, what i did wrong.
copy paste again the part from 334 to 350 into, then removed the D: ....
Egroupware is now working - google i must test .,..
i keep you up2date ... testing ongoing ...
thx cu Erwin
Attached patch Fix - v2 (obsolete) β€” β€” Splinter Review
Ok, this should do it for 2.6.1. For 2.7 or later we should go a step further and clean out all the old compatibilities for the drafts. Doing so means at least:

* Removing the draft 0-3 comments and sync-response tag handling
* Add support for 507 insufficient storage for paging requests
* Be sure to error out if the only response is a 507
* Make sure we are using the sync token in If-Headers, see Section 5

The document in question is: http://tools.ietf.org/html/rfc6578
Attachment #810335 - Attachment is obsolete: true
Attachment #812226 - Flags: review?(matthew.mecca)
I applied fix v2 and can successfully connect to Google again - so far the patch seems to be correct.

But not all events from the Google Calendar are synchronized to Lightning. I only can see events which I imported to Google more than two years ago when changing from an old Symbian phone to Android. All events I created since then (via phone or via Lightning) are currently not shown in Lightning (but I still can see them on the phone, of cause).

Does anyone have an idea if this has something to do with that sync-level-stuff or if I should open a new bug for this?
this is not related to that bug

this is as caldav is requesting data.
there is also still:

Zeitstempel: 30.09.2013 22:35:20
Fehler: [Exception... "'JavaScript component does not have a method named: "onAlarmsLoaded"' when calling method: [calIAlarmServiceObserver::onAlarmsLoaded]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "JS frame :: resource://calendar/modules/calUtils.jsm -> file:///C:/Users/Camel/AppData/Roaming/Thunderbird/Profiles/9dvislam.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calUtils.js :: notifyFunc :: line 1265"  data: no]
STACK: undefined
Quelldatei: resource://calendar/modules/calUtils.jsm -> file:///C:/Users/Camel/AppData/Roaming/Thunderbird/Profiles/9dvislam.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calUtils.js
Zeile: 1267

and also invites not working :(
Hi, average user here. Can someone upload the 'fixed' calDavRequestHandlers.js? So i can change the file on my pc.

I'm not a javascript coder and have no idea where to add the changes i see here. I tried using an online javascript editor but failed..
(In reply to zeep from comment #32)
> Hi, average user here. Can someone upload the 'fixed'
> calDavRequestHandlers.js? So i can change the file on my pc.
> 
> I'm not a javascript coder and have no idea where to add the changes i see
> here. I tried using an online javascript editor but failed..

Not hard.
1. Close Thunderbird
2. Open File Explorer/ or Search. 
3. Search for "calDavRequestHandlers.js"  on your computer
4. Open this file with Wordpad or Editor...what you prefer 
5. Open: https://bugzilla.mozilla.org/attachment.cgi?id=812226&action=diff
6. Copy the green line at the correct possition and delete the red line in the "calDavRequestHandlers.js" file you already opened with editor/wordpad/...
7. Save the "calDavRequestHandlers.js" File
8. Open Thunderbird and snyc, now should the calendar work again
(In reply to Marc from comment #33)
> (In reply to zeep from comment #32)
> > Hi, average user here. Can someone upload the 'fixed'
> > calDavRequestHandlers.js? So i can change the file on my pc.
> > 
> > I'm not a javascript coder and have no idea where to add the changes i see
> > here. I tried using an online javascript editor but failed..
> 
> Not hard.
> 1. Close Thunderbird
> 2. Open File Explorer/ or Search. 
> 3. Search for "calDavRequestHandlers.js"  on your computer
> 4. Open this file with Wordpad or Editor...what you prefer 
> 5. Open: https://bugzilla.mozilla.org/attachment.cgi?id=812226&action=diff
> 6. Copy the green line at the correct possition and delete the red line in
> the "calDavRequestHandlers.js" file you already opened with
> editor/wordpad/...
> 7. Save the "calDavRequestHandlers.js" File
> 8. Open Thunderbird and snyc, now should the calendar work again

That's what i tried to do before posting. The thing is, when i open calDavRequestHandlers.js (in an online javascript editor that i found, http://js.do/), it only has 332 lines. So i'm missing a bunch of lines to begin with.
Didn't want to hyjack the discussion with my problem as to why is it not complete, so i figured ask for a complete calDavRequestHandlers.js.
Don't care about the lines. Just search for the lines before or after to find the postition. 
Try to use win editor oder wordpad, both works fine. I prefer Wordpad cuz of the layout.
I got it. Was editing the wrong calDavRequestHandlers.js.
Silly me.

Now having the same problem as Comment 29
https://bugzilla.mozilla.org/show_bug.cgi?id=920755#c29
    Hi folks,

        I tried these steps on my setup.  However, after restarting, I see the following error from Lightning:

    Error: 'Component is not available' when calling method: [nsIActivityManager::removeActivity] = NS_ERROR_NOT_AVAILABLE
    Source file: resource://app/modules/activity/autosync.js
    Line: 203


    It syncs up to a point (about 2 years ago ;-)) but then it stops!  Looks like the same issues at zeep above and Comment 29.

    Thanks

    Ciao
    -- Murali
Attachment #812226 - Flags: review?(matthew.mecca) → review+
Comment on attachment 812226 [details] [diff] [review]
Fix - v2

With this patch, I am able to successfully sync in Lightning 2.9a1. Though Google calendar is the only CalDAV instance I have to test against.

As an aside, I am not experiencing the problem mentioned in Comment 29. I am seeing events that I created yesterday using https://www.google.com/calendar
I'm using Lightning 2.6b2 with Thunderbird 24.  Not sure if that is the cause.  zeep and Christoph can chime in about their versions.

Do you guys think this is related?  Without it I cannot sync my calendar.
Just another data point:  I use multiple google calendars under one account.  I use no other calendars.  I was running 17.0.8 / 1.9.1, and just got updated to 24.0 / 2.6.  Fix v1 and fix v2 worked for me before the upgrade, and fix v2 is working for me after the upgrade.

Thanks.
not working on EGW.
i tried to install 2.9a1 with thunderbird 24 (mod the XPI), but lightning it is really not supported then (views, etc..)

so, i installed now again lightning 2.6 and then egroupware caldav again working.
(and google caldav not - as known)

now testing the patch again....
(In reply to Murali Balasubramaniam from comment #37)
>     Hi folks,
> 
>         I tried these steps on my setup.  However, after restarting, I see
> the following error from Lightning:
> 
>     Error: 'Component is not available' when calling method:
> [nsIActivityManager::removeActivity] = NS_ERROR_NOT_AVAILABLE
>     Source file: resource://app/modules/activity/autosync.js
>     Line: 203
> 
> 
>     It syncs up to a point (about 2 years ago ;-)) but then it stops!  Looks
> like the same issues at zeep above and Comment 29.
> 
>     Thanks
> 
>     Ciao
>     -- Murali

I tried it again and I see that my secondary calendar syncs up fine, but my primary calendar only syncs up until May 2011.  Any ideas?

Thanks

Ciao
-- Murali
I appreciate your testing, but please get the version numbers matched up correctly. Thunderbird 24 will only work with Lightning 2.6. If you are using an earlier (beta) or later (2.7+) version, please make sure you are using 2.6. This should clear out most of the issues mentioned.

For those of you having the problem with only seeing a specific range of events, do you have the Cache enabled (Offline Support in the Properties dialog)? If so, please try to disable+restart+enable+restart for that calendar.
Applied and testet patch v1 and v2.
On 2 Systems:
a) Linux (openSUSE 12.3 - i686, fully patched)
b) Windows XP (fully patched)

I have/sync 7 Google calendars with one account.

I have similar results as in comment 29

- At a new start of TB, the triangles are now gone.
- Synchronisations partially works. However, not complete.
  On several calendars major parts are missing.

-> The "Basic" calender (the first one with the e-mail address as calender ID) only syncs some events and only up to mid JAN 2011. Afterwards no events are shown.
-> Another calendar has some annual events up to now. But others up to JAN 2011.

I also deleted the Passwords in the Password Manager.
I removed the calendar(s) and set them up new.

Still the same.


Testet on both systems.
Testet with Offline-Mode (Cache) ON and OFF.
(In reply to Philipp Kewisch [:Fallen] from comment #43)
> I appreciate your testing, but please get the version numbers matched up
> correctly. Thunderbird 24 will only work with Lightning 2.6. If you are
> using an earlier (beta) or later (2.7+) version, please make sure you are
> using 2.6. This should clear out most of the issues mentioned.
> 
> For those of you having the problem with only seeing a specific range of
> events, do you have the Cache enabled (Offline Support in the Properties
> dialog)? If so, please try to disable+restart+enable+restart for that
> calendar.

Hi Philipp,

    I'm using Thunderbird 24 and Lightning 2.6b2.  Is there a non-beta version of Lightning 2.6?

Also, I do not have offline mode enabled now.  I used to have it in the past, but recently since I saw the issue with CalDav sync I have disabled it.  Still I do not see this issue.

Looks like this issue is present in an number of cases now.  4 folks, including me, have reported it.

If it helps I can gather more error logs.  Let me know.

Thanks

Ciao
-- Murali
I tested patch v1 with four google calendars in one google account, which seems to work for me. All dates seem to be synced.
I also have four icloud calendars in one icloud account which do no work when applying patch v1. I get an error 400 then. Removing patch v1 makes the icloud calendars work again, but then the google calendars stop working with error 400.
I use thunderbird 24 with lightning 2.6.
(In reply to Murali Balasubramaniam from comment #45)
> (In reply to Philipp Kewisch [:Fallen] from comment #43)
> > I appreciate your testing, but please get the version numbers matched up
> > correctly. Thunderbird 24 will only work with Lightning 2.6. If you are
> > using an earlier (beta) or later (2.7+) version, please make sure you are
> > using 2.6. This should clear out most of the issues mentioned.
> > 
> > For those of you having the problem with only seeing a specific range of
> > events, do you have the Cache enabled (Offline Support in the Properties
> > dialog)? If so, please try to disable+restart+enable+restart for that
> > calendar.
> 
> Hi Philipp,
> 
>     I'm using Thunderbird 24 and Lightning 2.6b2.  Is there a non-beta
> version of Lightning 2.6?
> 
> Also, I do not have offline mode enabled now.  I used to have it in the
> past, but recently since I saw the issue with CalDav sync I have disabled
> it.  Still I do not see this issue.
> 
> Looks like this issue is present in an number of cases now.  4 folks,
> including me, have reported it.
> 
> If it helps I can gather more error logs.  Let me know.
> 
> Thanks
> 
> Ciao
> -- Murali

I meant to say I still see the issue with offline mode disabled.

Thanks

Ciao
-- Murali
Hi folks,

   Are we required to add both patches V1 and V2? The only patch I applied was V2, since I thought it obsoleted V1.

Let me know.

Thanks

Ciao
-- Murali
for newbies ....

you need to do the changes from patch V2 (in V2 patch, there is V1 included).

patch parts ... see below

you see there is 1 line with a "+" at beginning, that means -> you must add this line
(you see also the where this line - should be line 342)
you see there is 1 line with a "-" at beginning, that means -> you must add this line
(you see also the where this line - should be line 358)

the leading "+" or leading "-" should not be added.


# HG changeset patch
# Parent 1ab1075f0883fefc0fc32963fe69fb129f17479e
diff --git a/calendar/providers/caldav/calDavRequestHandlers.js b/calendar/providers/caldav/calDavRequestHandlers.js
--- a/calendar/providers/caldav/calDavRequestHandlers.js
+++ b/calendar/providers/caldav/calDavRequestHandlers.js
@@ -339,6 +339,7 @@
           xmlHeader +
           '<sync-collection xmlns="DAV:">' +
             syncTokenString +
+            '<sync-level>1</sync-level>' +
             '<prop>' +
               '<getcontenttype/>' +
               '<getetag/>' +
@@ -355,7 +356,6 @@
                                                         queryXml,
                                                         "text/xml; charset=utf-8",
                                                         this.calendar);
-        httpchannel.setRequestHeader("Depth", "1", false);
         httpchannel.requestMethod = "REPORT";
         // Submit the request
         httpchannel.asyncOpen(this, httpchannel);
sorry, a copy/paste mistake ..

for newbies ....

you need to do the changes from patch V2 (in V2 patch, there is V1 included).

patch parts ... see below

you see there is 1 line with a "+" at beginning, that means -> you must add this line
(you see also the where this line - should be line 342)
you see there is 1 line with a "-" at beginning, that means -> you must remove this line
(you see also the where this line - should be line 358)

the leading "+" or leading "-" should not be added.
Hi camro,

    That's exactly what I've done.  Still no dice.  I see that even the secondary google calendar does not sync up correctly!

Thanks

Ciao
-- Murali
Yes, there is no need to test v1 as it is obsolete. Patch v2 is the one to test. 

I'd appreciate if we could keep the support comments down here, a 50 comment bug is already quite a bit. If you are not able to apply the patch with the comments above, please wait until the patch is available in the product. Please also avoid "me too" type comments unless you can provide new information. I don't mean this to be controlling, but with this amount of comments it gets harder to filter out the information I need to fix the bug.

If you are experiencing an issue with some events not being shown please try the following:

* enable calendar.debug.log and calendar.debug.log.verbose in the advanced config editor (Options > Advanced > General > Config Editor), restart Thunderbird
* With logs enabled, re-enable the cache (see comment 43)
* Check the error console for messages. I am particularly interested if the response to the sync-collection request contains a tag with "507 Insufficient Storage".
* If you are unsure, copy the logs into a site like http://pastebin.mozilla.org/ and post the link here. There is no way without extra addons (console^2) to copy multiple messages at once.
(both Linux and WinXP (@ 2 Workstations), + TB24, LN 2.6 +patch v2):

Thought need to give it some more time to sync ALL the events.

Now after letting it "sit" for a while*, the calendars ALL are now disabled again, with yellow triangle.
*I guess it might happen after the configured 30 minutes calendar refresh time.

(Next step will enable logs and give some more feedback.)
Note that there is still bug 920230 open, which means your calendars will fail after a while. For this bug it is only important that they synchronize all events at the beginning.
Tb 24.0  Lightning 2.6 (no use of offline/cache funktion)

Used Fix V2 and I have the same problem as above, only a few events load after start. But if I use manual sync it loads all events.

errors: Zeitstempel: 01.10.2013 11:58:13
Warnung: getAttributeNodeNS() sollte nicht mehr verwendet werden. Verwenden Sie stattdessen getAttributeNS().
Quelldatei: resource://calendar/modules/calXMLUtils.jsm
Zeile: 32

translation:(...shouldn't be used. Please use instead.... )
(In reply to Philipp Kewisch [:Fallen] from comment #54)
> Note that there is still bug 920230 open, which means your calendars will
> fail after a while. For this bug it is only important that they synchronize
> all events at the beginning.

Guess it is the same.
Because it never reaches the state where it has full synchronization.

Then it reaches the state (of de-activated or in-accessable calendars) as it startet in this bug.
As I mentioned, please give me links to debug logs if you are having the problem of partial syncs. Please don't paste it in a comment.

The warning about getAttributeNS is normal and can be disregarded.
I tested with patch v2, syncing with google works, syncing with icloud does not work.
Warning: CalDAV: Error doing webdav sync: 400
Aaah fuck. Well so much for that. Back to the drawing board.
(In reply to Philipp Kewisch [:Fallen] from comment #52)
> Yes, there is no need to test v1 as it is obsolete. Patch v2 is the one to
> test. 
> 
> I'd appreciate if we could keep the support comments down here, a 50 comment
> bug is already quite a bit. If you are not able to apply the patch with the
> comments above, please wait until the patch is available in the product.
> Please also avoid "me too" type comments unless you can provide new
> information. I don't mean this to be controlling, but with this amount of
> comments it gets harder to filter out the information I need to fix the bug.
> 
> If you are experiencing an issue with some events not being shown please try
> the following:
> 
> * enable calendar.debug.log and calendar.debug.log.verbose in the advanced
> config editor (Options > Advanced > General > Config Editor), restart
> Thunderbird
> * With logs enabled, re-enable the cache (see comment 43)
> * Check the error console for messages. I am particularly interested if the
> response to the sync-collection request contains a tag with "507
> Insufficient Storage".
> * If you are unsure, copy the logs into a site like
> http://pastebin.mozilla.org/ and post the link here. There is no way without
> extra addons (console^2) to copy multiple messages at once.

Hi Philipp, et al,

   Sorry to send too many updates.  Will keep it to a minimum.  I got the logs, but pastebin is truncating the file down to 2481 lines whereas the actual log has 48899 lines (20 times as much)!  Does it help to see the partial logs?

Anyway, I do not see the 507 message in the full logs.

Let me know what I should do at this time.

And a big thanks is due to working so diligently on this issue.

Thanks

Ciao
-- Murali
With Fix - V2 i get this in my error console:

218. CalDAV: Reseting sync token because server returned status code: 401
219. CalDAV: send(https://apidata.googleusercontent.com/caldav/v2/name%40gmail.com/events/): <?xml version="1.0" encoding="UTF-8"?>
<sync-collection xmlns="DAV:"><sync-token/><D:sync-level>1</D:sync-level><prop><getcontenttype/><getetag/></prop></sync-collection>
220. CalDAV: webdav-sync Token: null
221. CalDAV: recv: 
222. CalDAV: onStopRequest: no reader
223. [JavaScript Warning: "CalDAV: Error doing webdav sync: 401"]
224. [JavaScript Warning: There has been an error reading data for calendar: Google. However, this error is believed to be minor, so the program will attempt to continue. Error code: DAV_REPORT_ERROR: DAV_REPORT_ERROR. Description: There has been an error reading data for calendar: https://apidata.googleusercontent.com/caldav/v2/name@gmail.com/events. It has been disabled until it is safe to use it."]
225. [JavaScript Warning: "There has been an error reading data for calendar: Google. However, this error is believed to be minor, so the program will attempt to continue. Error code: READ_FAILED. Description: "]
226. CalDAV: recv: 
227. CalDAV: onStopRequest: no reader
Same here but the most recent event I can see in my primary calendar is August 21, 2010.  My secondary calendar also works fine as in other reports.

Thunderbird 24 newly upgraded, Lightning 2.6, with the additional line / removed line from the patch posted above.  No errors or warnings in the console.  Caching turned off.

This recovers from having yellow triangles on both calendars, but we're not quite there yet.

(In reply to Murali Balasubramaniam from comment #42)
> (In reply to Murali Balasubramaniam from comment #37)
> >     Hi folks,
> > 
> >         I tried these steps on my setup.  However, after restarting, I see
> > the following error from Lightning:
> > 
> >     Error: 'Component is not available' when calling method:
> > [nsIActivityManager::removeActivity] = NS_ERROR_NOT_AVAILABLE
> >     Source file: resource://app/modules/activity/autosync.js
> >     Line: 203
> > 
> > 
> >     It syncs up to a point (about 2 years ago ;-)) but then it stops!  Looks
> > like the same issues at zeep above and Comment 29.
> > 
> >     Thanks
> > 
> >     Ciao
> >     -- Murali
> 
> I tried it again and I see that my secondary calendar syncs up fine, but my
> primary calendar only syncs up until May 2011.  Any ideas?
> 
> Thanks
> 
> Ciao
> -- Murali
I installed fix version 2, thanks to comment 33. I think I did it correctly, since now every time I run Thunderbird and open lightning the sync with my google calendar is working both read and write. Which did not happen before the fix.

However, As time passes I am experiencing the usual triangle and message "temporarily not available" after being connected for about one hour. The only way to recover the calendar connection is to restart thunderbird.

Maybe this was already pointed out by others, in this case I am sorry, comments are so numerous and dense that I have given up reading them all thoroughly.
I'm having the same experience as Andy.

Patch is applied and it has restored "normal" functionality - but after a while the connection is lost. When re-trying there is no feedback in the error console whatsoever.

However - if I edit the calendar and change the update schedule, then save, I am re-prompted for the authorization and the calendar reconnects. Now, at this point, if I look at the saved passwords, there is a new entry for each new authorization. It seems the authorization is expiring on a regular basis and the only way to get it back is to re-authorize, which happens as Lightning starts up.

Hopefully this helps. It seems the only thing now is this loss of authorization.
Patch v2 works fine on Lightning 2.6 / Thunderbird 24.0 / Win7Pro SP1 (x64).

(In reply to louis from comment #65)
> I'm having the same experience as Andy.
> 
> Patch is applied and it has restored "normal" functionality - but after a
> while the connection is lost. When re-trying there is no feedback in the
> error console whatsoever.
> 
> However - if I edit the calendar and change the update schedule, then save,
> I am re-prompted for the authorization and the calendar reconnects. Now, at
> this point, if I look at the saved passwords, there is a new entry for each
> new authorization. It seems the authorization is expiring on a regular basis
> and the only way to get it back is to re-authorize, which happens as
> Lightning starts up.
> 
> Hopefully this helps. It seems the only thing now is this loss of
> authorization.

That is another bug. See bug 920230.
I'm going to go ahead and push this patch and open a new issue for not all items being transferred. As the release builds for 2.6.1 will be created tomorrow, we might have to postpone that to 2.6.2.

I believe the issue with not all events being synchronized is either a server issue, or something that has to do with the chunked downloading.
Attached patch Fix - v3 β€” β€” Splinter Review
Oh right, forgot to upload before: this is the current patch, it handles both icloud and google caldav.
Assignee: matthew.mecca → philipp
Attachment #812226 - Attachment is obsolete: true
Attachment #814547 - Flags: review+
Attachment #814547 - Flags: approval-calendar-release+
Attachment #814547 - Flags: approval-calendar-beta+
Attachment #814547 - Flags: approval-calendar-aurora+
Comment on attachment 814547 [details] [diff] [review]
Fix - v3

In my version of calDavRequestHandlers.js there is no channel.setRequestHeader(...), it's as follows:

        httpchannel.setRequestHeader("Depth", "1", false);
        httpchannel.requestMethod = "REPORT";
        // Submit the request
        httpchannel.asyncOpen(this, httpchannel);

In fact the entire surrounding block looks very different.  Is it possible this is the root of different behaviors as people patch different versions of the software?
Its fine. The patch applies on another recent patch to fix the re-authentication issues.
If I don't want to wait until the update is rolled out and also don't want to fiddle with the file on my own, where is the repository, where I can download the fixed calDavRequestHandlers.js?
Yeah, I'd also like to download just a file that replaces another one. The changes I made didn't work as  I can't dismiss reminders for example.
Phillip, can you provide the order to apply patches so it works?  I also get patch reject files when applying the patch to a lightning version 2.6 download.  I grabbed the attachments from the dependencies shown in bug 921131.
Couldn't you simply make an update available for Thunderbird Add-Ons? That would be the easiest way.

Thunderbird Add-Ons manager says "No updates found.".

Regards,
Martin
An update is underway, awaiting review from the addons.mozilla.org team. Please see the "other versions" page if you want it now.
I have installed Lightning 0.2.6.1 from the "other versions" with Thunderbird 24.0.1 but I still get the following errors:

[JavaScript Warning: "CalDAV: Unexpected response, status: HTTP/1.1 507 Insufficient Storage, href: /caldav/v2/name@googlemail.com/events/"]

[JavaScript Error: "[calCachedCalendar] replay action failed: null, uri=https://apidata.googleusercontent.com/caldav/v2/name@googlemail.com/events, result=2147500037, op=[xpconnect wrapped calIOperation]"]

And because of this it doesn't work.
How to fix this?
Thanks
(In reply to Philipp Kewisch [:Fallen] from comment #76)
> An update is underway, awaiting review from the addons.mozilla.org team.
> Please see the "other versions" page if you want it now.

Also installed Lightning 2.6.1 with Thunderbird 24.0.1. But still getting following Errors:

Assert failed: unexepcted endBatch!
resource://calendar/modules/calUtils.jsm -> file:///C:/Users/......Line: 1029

as well the warning

[JavaScript Warning: "CalDAV: Unexpected response, status: HTTP/1.1 507 Insufficient Storage, href: /caldav/v2/name@googlemail.com/events/"]

Any idea?
Thx
Sorry forgot to copy that line of the error

2: [resource://calendar/modules/calProviderUtils.jsm:647] cPB_endBatch
(In reply to Frederik Niedernolte from comment #77)
(In reply to ppp_at from comment #78)

You got a different error from the Google Calendar server: 507 Insufficient Storage. Therefore it is a different problem as the one reported and fixed in this bug. I would suggest to file a new bug report for the different problem.
(In reply to ppp_at from comment #79)
> Sorry forgot to copy that line of the error
> 
> 2: [resource://calendar/modules/calProviderUtils.jsm:647] cPB_endBatch

same error here lightning 2.6.1 and TB 24.0.1 on ubuntu.
(I setup google calendar with caldav - provider for google calendar can't accept invitation yet)


Timestamp: 10/25/2013 01:48:54 PM
Error: Assert failed: unexepcted endBatch!
2: [resource://calendar/modules/calProviderUtils.jsm:647] cPB_endBatch
3: [resource://calendar/modules/calUtils.jsm -> file:///home/ryan/.thunderbird/oy9fbb5e.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calDavRequestHandlers.js:464] wH_endDocument
4: [resource://calendar/modules/calUtils.jsm -> file:///home/ryan/.thunderbird/oy9fbb5e.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calDavRequestHandlers.js:423] wSL_onStopRequest
5: [null:0] null

Source File: resource://calendar/modules/calUtils.jsm -> file:///home/ryan/.thunderbird/oy9fbb5e.default/extensions/%7Be2fda1a4-762b-4020-b5ad-a41df1933103%7D/calendar-js/calUtils.js
Line: 1029
I don't know what to do anymore. After getting Lightning working with my google calendar september 30th, edited calDavRequestHandlers.js with change mentioned here, it worked for about 3 weeks again. Albeit with entries appearing after 15 minutes.

Since about a week it stopped working again, momentarily not available with the yellow triangle.
Today i updated both Lightning and Thunderbird, didn't help.

Now what can i do to make this calendar work again?
The strange thing is, on my mac with iCal, no problems, on my iphone, no problems. It's just on windows??

Please help
(In reply to zeep from comment #83)
> I don't know what to do anymore. After getting Lightning working with my
> google calendar september 30th, edited calDavRequestHandlers.js with change
> mentioned here, it worked for about 3 weeks again. Albeit with entries
> appearing after 15 minutes.
> 
> Since about a week it stopped working again, momentarily not available with
> the yellow triangle.
> Today i updated both Lightning and Thunderbird, didn't help.
> 
> Now what can i do to make this calendar work again?
> The strange thing is, on my mac with iCal, no problems, on my iphone, no
> problems. It's just on windows??
> 
> Please help

Have same experience. 
You might be affected by  bug 930132.
I've been able to verify that is indeed the case for me by using the Error Console window in Thunderbird.
There was an update a few days ago but Lightning still does not work with https://www.google.com/calendar/dav/email@gmail.com/events. It doesn't work for weeks now. Do you know when it is fixed again? This year lightning didn't work most of the time, although it could be such a great tool. :-(
Flags: needinfo?(philipp)
I would say it has been working most of the year. Updates are always problematic, and third party changes even more. I hope 2.6.3 will fix the remaining two issues.
Flags: needinfo?(philipp)
Sorry, it wasn't meant this way. ;-)

Fact is, that your product is that good that there is not really an alternative. And that makes it even harder if calDav and Lightning doesn't work for weeks. Especially at the point when it comes to an error, I mostly have some open reminders that don't pop up anymore and may get forgotten. Maybe Mozilla should change there update policy to get updates rolled out faster.

Thanks for your support!
Martin
My google CalDAV calendars work, but I cannot switch my Yahoo CalDAV calendars on.

The calendar URL is of the form:
https://caldav.calendar.yahoo.com/dav/your_username/Calendar/your_calendar_name

I removed the calendars and tried to add them again, but Thunderbird
asks for the login/password repeatedly and does not save the credentials in
its password manager.

Thunderbird 24.0.1 on Ubuntu 13.04 (x86_64)
Lightning 2.6.2
#hautboy@50mail.com

I configured Yahoo Calendar in a Mac iCal, and the calendar address is not as you typed. I don't know if this can help, but try it: 

https://caldav.calendar.yahoo.com//principals/users/your_username/

In iCal you don't need to specify the calendar name, it shows every calendar on that account. However, it gives me some errors in iCal, but if I try in BusyCal, it works without any problem. I don't use Lightning now, but I did it some time ago.

Best regards.

Guillermo, from Spain
For more info, please read next link: http://dmfs.org/wiki/index.php?title=Yahoo

Good luck.
Thanks. After another change of password, CalDAV sync seems to work again in Lightning.
This seems to be a Yahoo problem.
My post was off-topic anyway because this thread is about OAuth2. Sorry for that.
Hi, is this still an open bug? Since many weeks we cannot synchronize the calendar with our IPhone via Google anymore. I have tried all kind of patches and was still expecting an upgrade. So, if it is solved, it does not work for us.
Is there any workaround? Does anybody know a different solution to synchronize with my IPhones?
Management is pushing an exchange server, but I do not want to go there if any possible,
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: