Intermittent test_clock_set_alarm_snooze.py test_clock_set_alarm_snooze.TestClockSetAlarmSnooze.test_clock_set_alarm_snooze | AssertionError: '15 minutes' != u'5 minutes'

RESOLVED FIXED

Status

RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: RyanVM, Assigned: zcampbell)

Tracking

({intermittent-failure})

unspecified
x86_64
Linux
intermittent-failure

Firefox Tracking Flags

(b2g-v1.2 affected)

Details

Attachments

(2 attachments, 3 obsolete attachments)

(Reporter)

Description

5 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=29835733&tree=B2g-Inbound

b2g_ubuntu64_vm b2g-inbound opt test gaia-ui-test on 2013-10-29 06:49:09 PDT for push 0f20f1d04204
slave: tst-linux64-ec2-397

06:58:12     INFO -  TEST-START test_clock_set_alarm_snooze.py
06:58:12     INFO -  test_clock_set_alarm_snooze (test_clock_set_alarm_snooze.TestClockSetAlarmSnooze)
06:58:26     INFO -  Modify the alarm snooze ... FAIL
06:58:26     INFO -  ======================================================================
06:58:26     INFO -  FAIL: None
06:58:26     INFO -  ----------------------------------------------------------------------
06:58:26     INFO -  Traceback (most recent call last):
06:58:26     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette_test.py", line 143, in run
06:58:26     INFO -      testMethod()
06:58:26     INFO -    File "/builds/slave/test/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/clock/test_clock_set_alarm_snooze.py", line 29, in test_clock_set_alarm_snooze
06:58:26     INFO -      self.assertEqual("15 minutes", new_alarm.alarm_snooze)
06:58:26     INFO -  TEST-UNEXPECTED-FAIL | test_clock_set_alarm_snooze.py test_clock_set_alarm_snooze.TestClockSetAlarmSnooze.test_clock_set_alarm_snooze | AssertionError: '15 minutes' != u'5 minutes'
06:58:26     INFO -  ----------------------------------------------------------------------
06:58:26     INFO -  Ran 1 test in 13.934s
06:58:26     INFO -  FAILED (failures=1)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Comment 16

5 years ago
Can we just disable this test?
Created attachment 826932 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13360

Pointer to Github pull-request
Attachment #826932 - Flags: review?(jlal)
We are going to xfail this on master until we can figure out why it is failing so often.
Created attachment 826945 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13361

Pointer to Github pull-request
Comment on attachment 826945 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13361

Disable instead of xfail
Attachment #826945 - Flags: review?(jlal)
(Assignee)

Comment 21

5 years ago
RyanVM/James Lal can you put some heat on the Clock app owners? This test fails regularly on Travis, too.

This test definitely used to be stable until about 1 week ago when something regressed it.
disabled on master (tbpl) -> https://github.com/mozilla-b2g/gaia/commit/8e502025aa80f641703dedcfb79b3117fba73b4e

Mike can you find and owner for this?
Flags: needinfo?(mike)
Attachment #826932 - Flags: review?(jlal) → review+
Attachment #826945 - Flags: review?(jlal) → review+
Evelyn: would you mind looking into this?
Assignee: nobody → evelyn
Flags: needinfo?(mike)
(In reply to Mike Pennisi [:jugglinmike] from comment #23)
> Evelyn: would you mind looking into this?

Yep.  I will try again soon when npm is back up and running.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Comment 27

5 years ago
Gonna have a short bash at this today myself.
(Assignee)

Comment 28

5 years ago
I think I have solved this but I need another hour or two and have run out of time for today. The solution will impact a lot of other tests so need to tread carefully.

Basically it is trying to select the option in the select wrapper before it is fully loaded.

I am trialling it here: https://github.com/mozilla-b2g/gaia/pull/13430
(Assignee)

Comment 29

5 years ago
Created attachment 828633 [details] [review]
github pr
Attachment #826932 - Attachment is obsolete: true
Attachment #826945 - Attachment is obsolete: true
Attachment #828633 - Flags: review?(florin.strugariu)
Attachment #828633 - Flags: review?(dave.hunt)
Attachment #828633 - Flags: review?(bob.silverberg)
Assignee: evelyn → zcampbell
(In reply to Zac C (:zac) from comment #28)
> I think I have solved this but I need another hour or two and have run out
> of time for today. The solution will impact a lot of other tests so need to
> tread carefully.
> 
> Basically it is trying to select the option in the select wrapper before it
> is fully loaded.
> 
> I am trialling it here: https://github.com/mozilla-b2g/gaia/pull/13430

Thank you for addressing this!  If you have any time, I have a couple of questions..

I had trouble getting the test to fail locally for me, what steps did you take to reproduce it? (versions of b2g, gaia, tbpl manifest, etc)

Also, how come the selector was an issue in this specific test only, and not for other tests that also used selectors in the alarm?
Attachment #828633 - Flags: review?(florin.strugariu) → review+
(Assignee)

Comment 31

5 years ago
(In reply to Evelyn Eastmond [:evhan55] from comment #30)
> (In reply to Zac C (:zac) from comment #28)
> > I think I have solved this but I need another hour or two and have run out
> > of time for today. The solution will impact a lot of other tests so need to
> > tread carefully.
> > 
> > Basically it is trying to select the option in the select wrapper before it
> > is fully loaded.
> > 
> > I am trialling it here: https://github.com/mozilla-b2g/gaia/pull/13430
> 
> Thank you for addressing this!  If you have any time, I have a couple of
> questions..
> 
> I had trouble getting the test to fail locally for me, what steps did you
> take to reproduce it? (versions of b2g, gaia, tbpl manifest, etc)
> 
> Also, how come the selector was an issue in this specific test only, and not
> for other tests that also used selectors in the alarm?

Evelyn, I never could replicate it locally but I just had to try and predict what was happening. I did however use Travis to test some solutions; I pushed a branch to Travis which changed the Travis run script to run this test on a loop rather than the whole suite. Then I tried 3-4 different solutions. It was about a 1-2 in 50 failure so a really hard one!

Eventually by watching the test run locally, inspecting the code and going through the test case manually I came to this conclusion. I fixed two other intermittents yesterday that I couldn't replicate locally just by hacking away at it like this.

I'm not sure what contributing factors came to cause this test to fail alone but my experience of these UI tests are that timing, timing timing is everything. As the test and the Gaia app are essentially asynchronous to each other these kind of races happen often amongst the performance improvements and regressions.
Wow, Zax--that's a lot of effort. I really appreciate you taking the time to work through all this! Thanks also for the explanation. I'm struggling with race conditions for the JavaScript integration tests over on bug 907177, so it's nice to know I'm not alone :)
(Assignee)

Comment 33

5 years ago
Well I think it only took me an hour or two all up but it is my job so I have some practice!
Comment on attachment 828633 [details] [review]
github pr

Ideally we'd find a way to do this without locating an element by the text content, but I'm okay with this fix. Added a few comments that would be good to address in this patch too.
Attachment #828633 - Flags: review?(dave.hunt) → review-
(Assignee)

Comment 35

5 years ago
(In reply to Dave Hunt (:davehunt) from comment #34)
> Comment on attachment 828633 [details] [review]
> github pr
> 
> Ideally we'd find a way to do this without locating an element by the text
> content, but I'm okay with this fix. Added a few comments that would be good
> to address in this patch too.

I think there's pretty much no choice because of the way it rewrites the select element the text is the only thing in common. I guess, yes ideally Firefox OS would transpose something more robust than just the text.

It's also frustrating that there is no "loaded" indicator so locating by index could be risky.
(Assignee)

Comment 36

5 years ago
Created attachment 829361 [details] [review]
github pr 2

Dave's comments addressed. Re-review please
Attachment #828633 - Attachment is obsolete: true
Attachment #828633 - Flags: review?(bob.silverberg)
Attachment #829361 - Flags: review?(florin.strugariu)
Attachment #829361 - Flags: review?(dave.hunt)
Attachment #829361 - Flags: review?(bob.silverberg)
Comment on attachment 829361 [details] [review]
github pr 2

Would like to see bugs raised for creating an app object for FTU and revisiting the select wrapper when there are non-displayed characters. Otherwise, r+ from me.
Attachment #829361 - Flags: review?(dave.hunt) → review+
Why was this merged with review requests outstanding?
(Assignee)

Comment 40

5 years ago
Comment on attachment 829361 [details] [review]
github pr 2

Clear review requests
Attachment #829361 - Flags: review?(florin.strugariu)
Attachment #829361 - Flags: review?(bob.silverberg)
Created attachment 831683 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13671

Pointer to Github pull-request
Comment on attachment 831683 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13671

Uplifting to v1.2
Attachment #831683 - Flags: review?(trifandreialin)
Attachment #831683 - Flags: review?(moz.teodosia)
Attachment #831683 - Flags: review?(florin.strugariu)
Attachment #831683 - Flags: review?(andrei.hutusoru)
Comment on attachment 831683 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13671

r+
Attachment #831683 - Flags: review?(trifandreialin) → review+
Attachment #831683 - Flags: review?(florin.strugariu) → review+

Updated

5 years ago
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.