Closed Bug 974554 Opened 10 years ago Closed 10 years ago

[B2G][Calendar][Yahoo!] Importing Yahoo Calendar does not import pre-existing events

Categories

(Firefox OS Graveyard :: Gaia::Calendar, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(blocking-b2g:1.3+, b2g-v1.3 fixed, b2g-v1.3T fixed, b2g-v1.4 fixed, b2g-v2.0 fixed, b2g-v2.0M fixed, b2g-v2.1 affected, b2g-v2.2 affected)

VERIFIED FIXED
1.4 S6 (25apr)
blocking-b2g 1.3+
Tracking Status
b2g-v1.3 --- fixed
b2g-v1.3T --- fixed
b2g-v1.4 --- fixed
b2g-v2.0 --- fixed
b2g-v2.0M --- fixed
b2g-v2.1 --- affected
b2g-v2.2 --- affected

People

(Reporter: rkunkel, Assigned: gaye)

References

Details

(Whiteboard: permafail [p=8])

Attachments

(3 files)

Attached file logcat
Description:
When the user imports their Yahoo! Calendar to their device, the events they have created will not always be imported. 

Repro Steps:
1) Update Buri to v1.3 BuildID: 20140218004003
2) Open the Calendar app
3) Import a Yahoo! Calendar
4) Under Calendar settings, change Sync Calendar to 'Manually'
4) Exit and re-sync the calendar

Actual:
Yahoo calendar's are not imported or synced properly

Expected:
Yahoo calendar's can be imported and synced to see new changes and events

Environmental Variables:
Device: Buri 1.3 MOZ
BuildID: 20140218004003
Gaia: df60e9b49207d64da5647ab15760c122adabfba4
Gecko: b5afe0b10e93
Version: 28.0
Firmware Version: 1.2-device.cfg

Notes:
1) Sometimes it takes an unusually long amount of time to see the calendar events from a fresh sync.

Repro frequency: 5/7 (out of 7 testers, 5 were able to reproduce the issue)
See attached: logcat
I'm wondering if there's some server-side problem happening here.

Does this reproduce on 1.1? That'll confirm if that's true or not.
Keywords: qawanted
QA Contact: ckreinbring
It does repro on Buri 1.1 mozilla RIL.  My test account's calendar events are not seen on the Calendar app.

Build ID: 20140218041202
Gecko: https://hg.mozilla.org/releases/mozilla-b2g18/rev/d7f2811943d1
Gaia: c5cb252e5d1aa45d14f3a2ea182e73e2271e4496
Platform Version: 18.1
Firmware Version: V1.2-device.cfg
Keywords: qawanted
Whiteboard: burirun1.3-3 → burirun1.3-3, burirun 1.4-2
Whiteboard: burirun1.3-3, burirun 1.4-2 → permafail
So this appears to be a problem on Yahoo's side.

Dylan - Do you know who we can outreach to here to get this resolved on Yahoo's side? We're currently unable to import Yahoo Calendars, which apparently is a problem on Yahoo's end.
Flags: needinfo?(doliver)
Gareth has a couple of ideas, shifting ni? to him.
Flags: needinfo?(doliver) → needinfo?(gaye)
blocking-b2g: --- → 1.3?
Flags: needinfo?(gaye)
Gareth, can you give more details about what we'd be blocking on here? Is there a fix or workaround you have in mind that we can do on our side to resolve it?
Assignee: nobody → gaye
I'm going to work with some engineers who work on yahoo calendar to diagnose whether the client or server is protocol-incompatible. Here are the scenarios that can come out of that:

Client isn't to spec
====================

We should block 1.3 on fixing the client.

Server isn't to spec
====================

We should see how soon Yahoo can fix the issue. If it can't happen immediately, there may be some spec-compliant workaround we can hack together.

Nobody knows
============
IMHO we should block on a [small] patch to disable the yahoo specific calendar auth. Yahoo users would still be able to use their calendars to the extent that they can now by entering in the address for the caldav server (ie https://calendar.yahoo.com). This would be much better though since right now we are "advertising" yahoo as a working thing in the ui when it's actually broken.
cc: Adam & Wilfred -- possible that we'll have to consider dropping Yahoo calendar as an option from 1.3 if we can't get a fast enough investigation and resolution on why it stopped working.
We either need to resolve this or turn off Yahoo calendar support, so I'm blocking this for 1.3.
blocking-b2g: 1.3? → 1.3+
So https://gist.github.com/gaye/9979270 is an example caldav interaction with Yahoo. I think the issue is that they aren't namespacing their calendar-data elements under DAV: like everything else since our parser is failing to extract the calendar-data element. Hopefully this is a simple fix.
The fix would definitely be simple on our side, but I think that their responses may not be compliant given 9.6.  CALDAV:calendar-data XML Element in http://www.ietf.org/rfc/rfc4791.txt. I've sent along my diagnosis so I'll update when they get back to me.
I just tested this on v1.4 on a Peak.

Gaia: 26983f356ecb1bcf30e862d334b5de790071803e
Gecko: 686bc9b06eb59da83f77f48c690ba50ae6d16596

Using the manifest in B2G and flashing gaia with PRODUCTION=1 MOZILLA_OFFICIAL=1 OPTIMIZE_GAIA=1 make reset-gaia.

I tried both using the Yahoo! calendar option and the CalDav option with the Yahoo! caldav URL.  None of the existing contacts in my Yahoo! calendar get imported.
Since this appears to be a problem with the format of the data we are receiving from Yahoo and it affects all versions of FxOS, I recommend that we un-block and potentially WONTFIX this bug. 

Removing Yahoo support from 1.3 will do nothing to address current 1.0.1 and 1.1 users, and will make it more difficult for 1.3 users in the future assuming Yahoo does fix their implementation sometime soon.
blocking-b2g: 1.3+ → 1.3?
(In reply to Dylan Oliver [:doliver] from comment #14)
> Since this appears to be a problem with the format of the data we are
> receiving from Yahoo and it affects all versions of FxOS, I recommend that
> we un-block and potentially WONTFIX this bug. 

I don't think a WONTFIX is an option. The reality is that we've a broken feature active on all branches right now in the calendar.

What we need to do here is find a path to outreaching to Yahoo to fix this problem on their end. We probably don't need a block on a change on our side, but we do need to do something about this. We're going to pack on loads of support complaints if we sit there and do nothing.

I'm going to start a thread on drivers about this to see what we can do. I'll also ping the SUMO team to get an article up for this known issue.
blocking-b2g: 1.3? → ---
Maybe wontfix is the wrong semantic -- what I mean to say is that I don't know that we should be making any changes on our side because if Yahoo are able to restore the proper namespace then it will fix all versions. 

We have reached out to Yahoo and they are investigating.
It seems like if Yahoo broke this and didn't notice because other clients didn't break, then it's also appropriate to file an enhancement to make the parser more flexible.

But having said that, the calendar-data elements look properly namespaced.  Comment 11 referencing section 9.6 seems satisfied since the default namespace is configured to urn:ietf:params:xml:ns:caldav because the root namespace declaration via the attribute xmlns="urn:ietf:params:xml:ns:caldav".
But what about the xmlns:DAV="DAV:" attribute also in the root namespace? FTR I am not an xml expert...
So, while the URI of "DAV:" seems ridiculous, I guess it makes sense because "dav" is actually a URI scheme.

Certainly RFC4918 declaring WebDAV makes uses of it:
http://tools.ietf.org/html/rfc4918#section-4.3.1

For more context on the XML namespace stuff, check out http://en.wikipedia.org/wiki/XML_namespace#Namespace_declaration
So after some more reading and debugging I think that the issue is actually that we can't fetch the calendar home that gets returned via the DAV:href tag. The calendar-data element is definitely correctly namespaced.
Ok next guess is that we're choking parsing the xml comment thing inside the calendar-data element
Hi Miller,

Would you mind taking a look? Relevant things to read/consult are section 9.6 of the caldav rfc and https://gist.github.com/gaye/9979270 which is an example client server interaction between us and Yahoo!.
Attachment #8403681 - Flags: review?(mmedeiros)
Comment on attachment 8403681 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/caldav/pull/21

LGTM! nice work.
Attachment #8403681 - Flags: review?(mmedeiros) → review+
Here's the caldav part https://github.com/mozilla-b2g/caldav/commit/a1cfcdfcc5acfe9c233267073fe67d24cdee000c on master.

RyanVM/jhford: mozilla-b2g/caldav is a sub-repository of gaia so this part does *not* need to be uplifted. There is a gaia patch [coming in a couple minutes] which will need to be uplifted.
blocking-b2g: --- → koi?
All branches are affected ^^
We aren't triaging 1.2 bugs anymore.
blocking-b2g: koi? → ---
blocking-b2g: --- → 1.3?
blocking-b2g: 1.3? → 1.3+
https://github.com/mozilla-b2g/gaia/commit/70274690ae585a027dacc6c0e33f1c9181ad8325 merged on master
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: permafail → permafail [p=8]
Target Milestone: --- → 1.4 S5 (11apr)
Please request approval-gaia-v1.3? on this patch when it is ready for v1.3 uplift.
NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): Yahoo! made changes to their caldav server implementation which broke our interoperability. Their changes were kosher with regards to the protocol. Our client was not [entirely] to spec.
[User impact] if declined: Calendar users on 1.3 will not be able to sync their Yahoo! calendars, events, etc.
[Testing completed]: Tested manually, added some unit tests in caldav lib
[Risk to taking this patch] (and alternatives if risky): None that I am aware of.
[String changes made]: Strings are the same as I left them ;)
Attachment #8405123 - Flags: approval-gaia-v1.3?
[Environment]
Gaia      4bc61f9faa7fe76c911b4a7f3f89424cd38400bf
Gecko     https://hg.mozilla.org/mozilla-central/rev/83ae54e18689
BuildID   20140410160203
Version   31.0a1
ro.build.version.incremental=eng.archermind.20131114.105818
ro.build.date=Thu Nov 14 10:58:33 CST 2013

[Result]
PASS
Status: RESOLVED → VERIFIED
\o/
Attachment #8405123 - Flags: approval-gaia-v1.3? → approval-gaia-v1.3+
This issue occurs on Open C 1.4, should I open a new issue?

1.4 Environmental Variables:
Device: Open_c 1.4 MOZ
BuildID: 20140505000201
Gaia: fccb15d6940db51615545574877a62d69406b1c2
Gecko: bf937fbec8c1
Version: 30.0
Firmware Version: P821A10-ENG_20140410
(In reply to psiphantong from comment #34)
> This issue occurs on Open C 1.4, should I open a new issue?

Yes.
Hi Gareth,
A similar issue was just reported by partner per bug 1098670. Could you take a look and check whether it is same issue here? Thanks!
Flags: needinfo?(gaye)
See Also: → 1098670
I don't see a clear link between this issue and bug 1098670, cancelling ni here, will continue discussion over there.
Flags: needinfo?(gaye)
You need to log in before you can comment on or make changes to this bug.