1018833 - Calendar sync duplicates events

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::Calendar, defect)

Description

[Lucas Adamski [:ladamski]]

Over time I get more and more duplicate copies of events on my calendar.  Seems likely to be related network issues / timeouts, though impossible to test ATM due to bug 1018832

Comment 1

[Lucas Adamski [:ladamski]]

This is a long running bug, but reproduces on recent Flame 2.0 builds.

Comment 2

[Lucas Adamski [:ladamski]]

FWIW may be related to receiving updates to existing events.

Comment 3

[Gareth Aye [:gaye] (back from PTO)]

One detail that will help me narrow down the root cause is whether or not the duplicate event copies get persisted after you close and relaunch the calendar. If the duplicates do stick around, then there's a backend issue. Otherwise it's a UI bug.

Comment 4

[Lucas Adamski [:ladamski]]

Un-helpfully, both appear to be true.  One event (the triplicate) persists between app restarts but I don't see the others.  None of these are duplicated in the zimbra web UI tho.

Comment 5

[Gareth Aye [:gaye] (back from PTO)]

Update: I was able to reproduce the issue this morning after adding the public calendar's firefox os share. The calendar app picked up two calendars with the same displayname property. My next step is to look at the ical that Zimbra sends us which we are misinterpreting.

Comment 6

[Gareth Aye [:gaye] (back from PTO)]

I am actually curiously no longer able to reproduce this issue. Lucas - would you mind sharing with me the calendar / share that generates the duplicates? Going to tag qawanted for another look as well...

Comment 7

[Gareth Aye [:gaye] (back from PTO)]

It's also worth noting that we have lots of calendars and events at mozilla which look very similar and that Lucas and I may have been mistaken about there being an issue here. For instance, today I have (in both zimbra and fxos)

1. Firefox OS release triage at 1:00 am organized by dietrich which I assume is a TPE thing
2. Firefox OS release triage at 11:30 am organized by dietrich which is presumably a US/EUR thing
3. B2G triage at 11:30 am organized by preeti which I guess is the same as 2
4. B2G Weekly Triage at 11:30 am organized by publiccalendar and sent by :jsmith which is the same as 2 and 3

Comment 8

[Gareth Aye [:gaye] (back from PTO)]

For me, qawanted will always be a whiteboard tag.

Comment 9

[Lucas Adamski [:ladamski]]

I'm definitely still seeing this issue.  They persist between reboots, and don't exist on the zimbra web view or other apps/devices (ical, ipad, etc).

These are happening on my personal calendar not a public shared one so I'm not sure I want to share that.. I don't think its anything with the calendar itself as the duplication is inconsistent from devices to devices (I backed up the a profile and put it on a different phone, yet now each shows different duplicates).

I still suspect this bug is triggered by updates to events not being handled properly.

Comment 10

[Tony Chung [:tchung]]

Josh, can you help get some folks on reproducing this?   We also need a regression range once reproduced.  It's been identified that its quite intermittent to reproduce.   i would suggest different types of calendars like yahoo, google, exchange.

Comment 11

[Jason Smith [:jsmith]]

(In reply to Tony Chung [:tchung] from comment #10)
> Josh, can you help get some folks on reproducing this?   We also need a
> regression range once reproduced.  It's been identified that its quite
> intermittent to reproduce.   i would suggest different types of calendars
> like yahoo, google, exchange.

I don't think this is a good candidate for a window. Windows usually require consistent STR, since we need a reliable way to determine that we've got right root cause for the bug.

Comment 12

[Kate Glazko]

I can help with this.

Comment 13

[Joshua Mitchell (Inactive)]

Dixon - let's see if we can get them some consistent STRs

Comment 14

[Duane Dixon [:ddixon]]

STR

Pre-requisets: 
In Calendar App: add Google account and activate "Holidays in the United States" and "Offline Calendar".

1. In Calendar App: create at least 7 events over 7 consecutive days of the month. 
2. Power off device.
3. Power on and open Calendar App.

Actual Results: 
Saved events will be duplicated and/or shifted to different days.

Expected Results: 
All created calendar events save correctly.

Repro Frequency: 
8/10  about 80%

Environmental Variables:
Device: Flame Master
Build ID: 20140630182301
Gaia: bc3bbf42d2a606f6b7038881cff5ec3795fdf953
Gecko: a3af97c421d3
Version: 33.0a1 (Master)
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:33.0) Gecko/33.0 Firefox/33.0

Comment 15

[Joshua Mitchell (Inactive)]

I do not see that this has actually been proven to be a regression so let me get someone on some branch checks before I put the window-wanted tag back. 

QA-Wanted to check Flame 1.4 (and 1.3 if it repros on 1.4)

Comment 16

[Duane Dixon [:ddixon]]

Issue DOES occur in 1.3 Flame branch. 

Environmental Variables:
Device: Flame 1.3
Build ID: 20140616171114
Gaia: e1b7152715072d27e0880cdc6b637f82fa42bf4e
Gecko: e181a36ebafaa24e5390db9f597313406edfc794
Version: 28.0 (1.3)
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:28.0) Gecko/28.0 Firefox/28.0

Issue DOES occur in 1.4 Flame branch.

Environmental Variables:
Device: Flame 1.4
Build ID: 20140630064330
Gaia: aa896d5db1b4929f3bf31a0f4bb7de50530222a8
Gecko: 55b529051ef2
Version: 30.0 (1.4)
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:30.0) Gecko/30.0 Firefox/30.0

Comment 17

[Joshua Mitchell (Inactive)]

Issue is not a regression

Comment 18

[Gareth Aye [:gaye] (back from PTO)]

(In reply to Duane Dixon [:ddixon] from comment #14)
> STR
> 
> Pre-requisets: 
> In Calendar App: add Google account and activate "Holidays in the United
> States" and "Offline Calendar".
> 
> 1. In Calendar App: create at least 7 events over 7 consecutive days of the
> month. 
> 2. Power off device.
> 3. Power on and open Calendar App.
> 
> Actual Results: 
> Saved events will be duplicated and/or shifted to different days.
> 
> Expected Results: 
> All created calendar events save correctly.
> 
> Repro Frequency: 
> 8/10  about 80%
> 
> Environmental Variables:
> Device: Flame Master
> Build ID: 20140630182301
> Gaia: bc3bbf42d2a606f6b7038881cff5ec3795fdf953
> Gecko: a3af97c421d3
> Version: 33.0a1 (Master)
> Firmware Version: v122
> User Agent: Mozilla/5.0 (Mobile; rv:33.0) Gecko/33.0 Firefox/33.0

I know this is getting to the point of sillyness, but would you mind take a video of the repro so that we're all on the same page? I haven't been able to reproduce with gaia at 075f32fc0b411eadbd91bbc65e06dc5983a09ac9 (on 2.1).

Comment 19

[Duane Dixon [:ddixon]]

I followed my STR in commment 14 in the video here. 

Actual Results: The video illustrates that calendar events were created for the dates: August 1-7.  The events did not save where they were created (dates 1-7 were blank). 

Note: Skip to 1:00 in the video to bypass me signing into my Google account. 

Video Link: http://youtu.be/GLQty2PHmlA

Environmental Variables:
Device: Flame Master
Build ID: 20140701173553
Gaia: 85e97290431ce6aa0a965421e84d6070cd899129
Gecko: 7075808c3306
Version: 33.0a1 (Master)
Firmware Version: v122
User Agent: Mozilla/5.0 (Mobile; rv:33.0) Gecko/33.0 Firefox/33.0

Comment 20

[Gareth Aye [:gaye] (back from PTO)]

So I think the issue here is actually that users are still able to create events on calendars that they're hiding (since Duane toggled the visibility of his gcal at the beginning of the video). I think Holidays in the United States is a read-only dav collection, so I think we were actually writing the events to the gcal, but they weren't being shown since the calendar was hidden. For instance, if at the end of the video you untoggle your personal calendar, do the events show up?

Comment 21

[Duane Dixon [:ddixon]]

Gareth, I think you are correct.  

However, I was still able to reproduce the events I created to shift to different days after rebooting the device. 

Pre-requisets: 
In Calendar App: add Google account and activate: Gcal, Holidays in the United States, Contacts birthdays and Offline Calendar. 

1. In Calendar App: create at least 4 events over 4 consecutive days of the month. 
2. Restart off device.
3. Open Calendar App.

Actual Results: 
Saved events will be shifted to different days.

Expected Results: 
All created calendar events save correctly.

NEW video: http://youtu.be/X1p1sh7zBtY

Comment 22

[Dylan Oliver [:doliver]]

(In reply to Duane Dixon [:ddixon] from comment #21)
> Gareth, I think you are correct.  
> 
> However, I was still able to reproduce the events I created to shift to
> different days after rebooting the device. 

It looks like you have a time zone shift for some reason -- probably worth exploring further but I'd suggest moving that out to a different bug as we've meandered away from the initial description here.

Comment 23

[Tony Chung [:tchung]]

(In reply to Dylan Oliver [:doliver] from comment #22)
> (In reply to Duane Dixon [:ddixon] from comment #21)
> > Gareth, I think you are correct.  
> > 
> > However, I was still able to reproduce the events I created to shift to
> > different days after rebooting the device. 
> 
> It looks like you have a time zone shift for some reason -- probably worth
> exploring further but I'd suggest moving that out to a different bug as
> we've meandered away from the initial description here.

(In reply to Gareth Aye [:gaye] from comment #20)
> So I think the issue here is actually that users are still able to create
> events on calendars that they're hiding (since Duane toggled the visibility
> of his gcal at the beginning of the video). I think Holidays in the United
> States is a read-only dav collection, so I think we were actually writing
> the events to the gcal, but they weren't being shown since the calendar was
> hidden. For instance, if at the end of the video you untoggle your personal
> calendar, do the events show up?


Duane, your STR and bug seems to be a different bug than this one.  (given you toggled hide/show of the calendar).  can you file a new bug with your steps?   The original bug here didnt show the hide/showing of the gcal.   THanks.

Comment 24

[Duane Dixon [:ddixon]]

Bug submitted: 

Bug 1041815 - [B2G][Gaia][Calendar] Calendar events do not save correctly when gcal is toggled off and app is closed then reopened

https://bugzilla.mozilla.org/show_bug.cgi?id=1041815

Comment 25

[Dylan Oliver [:doliver]]

Unblocking for 2.0 due to the difficulty to reproduce and the apparent rarity of the bug. Working directly with Lucas last week showed that there does appear to be a problem with the iCal parser getting tripped up by some event(s) on his calendar. This is then putting the database into a bad state and resulting in the duplicate events on re-sync. To this point, however, we have been unable to identify any specific event that is causing the problem and so we need further digging (or a new reproducible case) to direct the investigation into the parser code.

Comment 26

[Adam Alldredge [:AdamA][Inactive]]

STR

This issue occurs with Yahoo, Caldav, and Google accounts. When a user has one online calendar set up, then signs into a second one with a repeating event that lasts a long time, the events will be duplicated in the calendar 5 months after the repeating event was added. If the user has three accounts signed into the phone the event will be duplicated 3 times. This issue also occurs if the repeating event is added after the calendar is added to the phone as long as it is not the first calendar account that was added.
Example: If a user adds a repeating event that starts on August 19th, 2014 and repeats indefinitely, the event will start being duplicated on January 20th, 2015.

Prerequisites: 
Have at least one online calendar account set up in the calendar app.

1. Update a Flame device to BuildID: 20140818040201
2. Add a repeating event that does not have an end date set to a calendar online
3. Add the calendar with the repeating event to the calendar app
4. Scroll through the calendar to approximately 5 months after the event was added.
5. Observe repeating event being duplicated

Actual Results: 
Repeating calendar events will be duplicated in the calendar 5 months after they begin.

Expected Results: 
Calendar events are not duplicated

Repro Frequency: 
10/10

Environmental Variables:
Device: Flame 2.1 Master (319mb)
BuildID: 20140818040201
Gaia: aa8aace12d65956dd9525da5dac66e0d3b28597f
Gecko: 0aaa2d3d15cc
Version: 34.0a1 (Master)
Firmware Version: v123
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Comment 27

[Adam Alldredge [:AdamA][Inactive]]

Attachment: logcat

Comment 28

[Adam Alldredge [:AdamA][Inactive]]

Attachment: Duplicated events

Comment 29

[Gareth Aye [:gaye] (back from PTO)]

Thanks Adam!

Comment 30

[Tony Chung [:tchung]]

[Blocking Requested - why for this release]:

I am hitting this on the 9/8 mozilla-aurora build.   My STR is basically add my calendars to the app, and a handful of my recurring events are all duplicated across the different week/month/day views.

Gaia      a8e4d26555e5713ec6c72270cfd0cfabc096a0d3
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/746f24f9d21d
BuildID   20140908000204
Version   34.0a2
ro.build.date   Fri Jun 27 15:57:58 CST 2014
ro.bootloader   L1TC00011230
ro.build.version.incremental   110

See screenshots.  reno'ming to see if we can get this fixed as its a pretty bad user experience.

Comment 31

[Tony Chung [:tchung]]

Attachment: day view

Comment 32

[Tony Chung [:tchung]]

Attachment: week view

Comment 33

[Tony Chung [:tchung]]

Attachment: month view

Comment 34

[Tony Chung [:tchung]]

per Productivity Triage, blocking.   Gareth, let me know how i can share my calendar with you.  FWIW, all the items that are duplicating seems to be recurring events.

Comment 35

[Miller Medeiros [:millermedeiros]]

I'm taking this since Gareth is swamped.

Comment 36

[Miller Medeiros [:millermedeiros]]

Just to sum up what I discovered today (still need to do more tests tho):

I was able to reproduce the bug with Yahoo and Zimbra calendars.. as :AdamA said on Comment #26, it looks like it only happens if you have multiple accounts (doesn't matter which one you add first). I have a test account with no events, and just by adding this account and moving any view 5-6 months in the "future" (today is Sep 24, so I moved past March 2nd 2015) I started seeing duplicate events..

What is really weird is that I don't think it matters what is the first occurrence of the repeating event, they always started to duplicate after March 2nd 2015 (190 days in the future, ~6mo). When I changed the system date to Dec 1st the duplicate events only started after May 8th (158 days in the future, ~5mo)

I'll keep investigating to try to understand what is going on, but at least I made some progress.

Comment 37

[Tony Chung [:tchung]]

Attachment: screenshot of notifications

FWIW, it affects notifications view also.

Comment 38

[Miller Medeiros [:millermedeiros]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/24572

problem was caused because it was calling `ensureRecurrencesExpanded` on the caldav provider multiple times (once for each account) before it was able to update the `lastRecurrenceId`.

unit tests should catch regressions.

Comment 39

[Miller Medeiros [:millermedeiros]]

Attachment: [v2.1] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/24769

separate PR for v2.1 branch to speed up the uplift process (specially since we might need to change things on master after AMD modules)

Comment 40

[Gareth Aye [:gaye] (back from PTO)]

Generally looks great to me. Let's do one more review pass after this.

Comment 41

[Gareth Aye [:gaye] (back from PTO)]

Comment on attachment 8499745 [details] [review]
[v2.1] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/24769

Looks good! Thanks Miller!

Comment 42

[Miller Medeiros [:millermedeiros]]

Comment on attachment 8497585 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/24572

basically same patch as v2.1, so carrying the r+ from other pull request.

Comment 43

[Miller Medeiros [:millermedeiros]]

Comment on attachment 8499745 [details] [review]
[v2.1] Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/24769

[Approval Request Comment]
[Bug caused by] (feature/regressing bug #): none
[User impact] if declined: calendar will show duplicate events/notifications if user has more than 1 calendar account
[Testing completed]: manual and unit
[Risk to taking this patch] (and alternatives if risky): low risk, changes are isolated
[String changes made]: none

Comment 44

[Miller Medeiros [:millermedeiros]]

landed on master: https://github.com/mozilla-b2g/gaia/commit/1025f0b3c5c6d95e522247909d469dbe6184d37d

Comment 45

[Ryan VanderMeulen [:RyanVM]]

v2.1: https://github.com/mozilla-b2g/gaia/commit/f352fd0ce0c6dd873128b44ea24cc0dd3e2fa5f6

Comment 46

[Roland Kunkel [:RolandK]]

Verified fixed for latest 2.1 and 2.2 Flame builds:

Enviromental Variables:
----------------------------------------
Device: Flame 2.1
BuildID: 20141011000201
Gaia: f5d4ff60ffed8961f7d0380ada9d0facfdfd56b1
Gecko: d813d79d3eae
Gonk: 52c909e821d107d414f851e267dedcd7aae2cebf
Version: 34.0a2 (2.1)
Firmware: V180
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Environmental Variables:
----------------------------------------
Device: Flame 2.2 Master
BuildID: 20141011040204
Gaia: 95f580a1522ffd0f09302372b78200dab9b6f322
Gecko: 3f6a51950eb5
Gonk: 52c909e821d107d414f851e267dedcd7aae2cebf
Version: 35.0a1 (2.2 Master)
Firmware: V180
User Agent: Mozilla/5.0 (Mobile; rv:35.0) Gecko/35.0 Firefox/35.0

Created online events that repeat without an end date are no longer duplicated.