Closed Bug 932491 Opened 11 years ago Closed 11 years ago

Intermittent test_clock_set_alarm_snooze.py test_clock_set_alarm_snooze.TestClockSetAlarmSnooze.test_clock_set_alarm_snooze | AssertionError: '15 minutes' != u'5 minutes'

Categories

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::UI Tests
Platform:
x86_64
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(b2g-v1.2 affected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
b2g-v1.2 --- affected

People

(Reporter: RyanVM, Assigned: zcampbell)

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 3 obsolete files)

github pr 2
46 bytes, text/x-github-pull-request
davehunt
: review+
Details | Review
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13671
358 bytes, text/html
Bebe
: review+
AlinT
: review+
AndreiH
: review+
Details 
https://tbpl.mozilla.org/php/getParsedLog.php?id=29835733&tree=B2g-Inbound

b2g_ubuntu64_vm b2g-inbound opt test gaia-ui-test on 2013-10-29 06:49:09 PDT for push 0f20f1d04204
slave: tst-linux64-ec2-397

06:58:12     INFO -  TEST-START test_clock_set_alarm_snooze.py
06:58:12     INFO -  test_clock_set_alarm_snooze (test_clock_set_alarm_snooze.TestClockSetAlarmSnooze)
06:58:26     INFO -  Modify the alarm snooze ... FAIL
06:58:26     INFO -  ======================================================================
06:58:26     INFO -  FAIL: None
06:58:26     INFO -  ----------------------------------------------------------------------
06:58:26     INFO -  Traceback (most recent call last):
06:58:26     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette_test.py", line 143, in run
06:58:26     INFO -      testMethod()
06:58:26     INFO -    File "/builds/slave/test/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/clock/test_clock_set_alarm_snooze.py", line 29, in test_clock_set_alarm_snooze
06:58:26     INFO -      self.assertEqual("15 minutes", new_alarm.alarm_snooze)
06:58:26     INFO -  TEST-UNEXPECTED-FAIL | test_clock_set_alarm_snooze.py test_clock_set_alarm_snooze.TestClockSetAlarmSnooze.test_clock_set_alarm_snooze | AssertionError: '15 minutes' != u'5 minutes'
06:58:26     INFO -  ----------------------------------------------------------------------
06:58:26     INFO -  Ran 1 test in 13.934s
06:58:26     INFO -  FAILED (failures=1)
Can we just disable this test?
Attachment #826932 - Flags: review?(jlal)
We are going to xfail this on master until we can figure out why it is failing so often.
Comment on attachment 826945 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13361

Disable instead of xfail
Attachment #826945 - Flags: review?(jlal)
RyanVM/James Lal can you put some heat on the Clock app owners? This test fails regularly on Travis, too.

This test definitely used to be stable until about 1 week ago when something regressed it.
disabled on master (tbpl) -> https://github.com/mozilla-b2g/gaia/commit/8e502025aa80f641703dedcfb79b3117fba73b4e

Mike can you find and owner for this?
Flags: needinfo?(mike)
Attachment #826932 - Flags: review?(jlal) → review+
Attachment #826945 - Flags: review?(jlal) → review+
Evelyn: would you mind looking into this?
Assignee: nobody → evelyn
Flags: needinfo?(mike)
(In reply to Mike Pennisi [:jugglinmike] from comment #23)
> Evelyn: would you mind looking into this?

Yep.  I will try again soon when npm is back up and running.
Gonna have a short bash at this today myself.
I think I have solved this but I need another hour or two and have run out of time for today. The solution will impact a lot of other tests so need to tread carefully.

Basically it is trying to select the option in the select wrapper before it is fully loaded.

I am trialling it here: https://github.com/mozilla-b2g/gaia/pull/13430
Attached file github pr (obsolete) — 

        Attachment #826932 -
        Attachment is obsolete: true

        Attachment #826945 -
        Attachment is obsolete: true

        Attachment #828633 -
        Flags: review?(florin.strugariu)

        Attachment #828633 -
        Flags: review?(dave.hunt)

        Attachment #828633 -
        Flags: review?(bob.silverberg)
Assignee: evelyn → zcampbell

    
    

    
  
(In reply to Zac C (:zac) from comment #28)
> I think I have solved this but I need another hour or two and have run out
> of time for today. The solution will impact a lot of other tests so need to
> tread carefully.
> 
> Basically it is trying to select the option in the select wrapper before it
> is fully loaded.
> 
> I am trialling it here: https://github.com/mozilla-b2g/gaia/pull/13430

Thank you for addressing this!  If you have any time, I have a couple of questions..

I had trouble getting the test to fail locally for me, what steps did you take to reproduce it? (versions of b2g, gaia, tbpl manifest, etc)

Also, how come the selector was an issue in this specific test only, and not for other tests that also used selectors in the alarm?

        Attachment #828633 -
        Flags: review?(florin.strugariu) → review+

    
    

    
  
(In reply to Evelyn Eastmond [:evhan55] from comment #30)
> (In reply to Zac C (:zac) from comment #28)
> > I think I have solved this but I need another hour or two and have run out
> > of time for today. The solution will impact a lot of other tests so need to
> > tread carefully.
> > 
> > Basically it is trying to select the option in the select wrapper before it
> > is fully loaded.
> > 
> > I am trialling it here: https://github.com/mozilla-b2g/gaia/pull/13430
> 
> Thank you for addressing this!  If you have any time, I have a couple of
> questions..
> 
> I had trouble getting the test to fail locally for me, what steps did you
> take to reproduce it? (versions of b2g, gaia, tbpl manifest, etc)
> 
> Also, how come the selector was an issue in this specific test only, and not
> for other tests that also used selectors in the alarm?

Evelyn, I never could replicate it locally but I just had to try and predict what was happening. I did however use Travis to test some solutions; I pushed a branch to Travis which changed the Travis run script to run this test on a loop rather than the whole suite. Then I tried 3-4 different solutions. It was about a 1-2 in 50 failure so a really hard one!

Eventually by watching the test run locally, inspecting the code and going through the test case manually I came to this conclusion. I fixed two other intermittents yesterday that I couldn't replicate locally just by hacking away at it like this.

I'm not sure what contributing factors came to cause this test to fail alone but my experience of these UI tests are that timing, timing timing is everything. As the test and the Gaia app are essentially asynchronous to each other these kind of races happen often amongst the performance improvements and regressions.

    
    

    
  
Wow, Zax--that's a lot of effort. I really appreciate you taking the time to work through all this! Thanks also for the explanation. I'm struggling with race conditions for the JavaScript integration tests over on bug 907177, so it's nice to know I'm not alone :)

    
    

    
  
Well I think it only took me an hour or two all up but it is my job so I have some practice!

    
    

    
  
Comment on attachment 828633 [details] [review]
github pr

Ideally we'd find a way to do this without locating an element by the text content, but I'm okay with this fix. Added a few comments that would be good to address in this patch too.

        Attachment #828633 -
        Flags: review?(dave.hunt) → review-

    
    

    
  
(In reply to Dave Hunt (:davehunt) from comment #34)
> Comment on attachment 828633 [details] [review]
> github pr
> 
> Ideally we'd find a way to do this without locating an element by the text
> content, but I'm okay with this fix. Added a few comments that would be good
> to address in this patch too.

I think there's pretty much no choice because of the way it rewrites the select element the text is the only thing in common. I guess, yes ideally Firefox OS would transpose something more robust than just the text.

It's also frustrating that there is no "loaded" indicator so locating by index could be risky.

    
    

    
  

      
      
      
      

        Attached file
          
        github pr 2
        — 
      

    


  
Dave's comments addressed. Re-review please

        Attachment #828633 -
        Attachment is obsolete: true

        Attachment #828633 -
        Flags: review?(bob.silverberg)

        Attachment #829361 -
        Flags: review?(florin.strugariu)

        Attachment #829361 -
        Flags: review?(dave.hunt)

        Attachment #829361 -
        Flags: review?(bob.silverberg)

    
    

    
  
Comment on attachment 829361 [details] [review]
github pr 2

Would like to see bugs raised for creating an app object for FTU and revisiting the select wrapper when there are non-displayed characters. Otherwise, r+ from me.

        Attachment #829361 -
        Flags: review?(dave.hunt) → review+

    
    

    
  
Why was this merged with review requests outstanding?

    
    

    
  
Comment on attachment 829361 [details] [review]
github pr 2

Clear review requests

        Attachment #829361 -
        Flags: review?(florin.strugariu)

        Attachment #829361 -
        Flags: review?(bob.silverberg)

    
    

    
  
Comment on attachment 831683 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13671

Uplifting to v1.2

        Attachment #831683 -
        Flags: review?(trifandreialin)

        Attachment #831683 -
        Flags: review?(moz.teodosia)

        Attachment #831683 -
        Flags: review?(florin.strugariu)

        Attachment #831683 -
        Flags: review?(andrei.hutusoru)

    
    

    
  
Comment on attachment 831683 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/13671

r+

        Attachment #831683 -
        Flags: review?(trifandreialin) → review+

        Attachment #831683 -
        Flags: review?(florin.strugariu) → review+

    
  
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: