966729 - Perma-failing UI test: test_privileged_app_contacts_prompt.py

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[Kevin Grandon :kgrandon]

This has started failing quite often now, causing the tree to be red  > 50% of the time.

test_privileged_app_contacts_prompt.py test_privileged_app_contacts_prompt.TestPrivilegedAppContactsPrompt.test_contact_prompt

test_contact_prompt (test_privileged_app_contacts_prompt.TestPrivilegedAppContactsPrompt) ... ERROR

======================================================================

ERROR: None

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/build/mozilla-b2g/gaia/travis_venv/local/lib/python2.7/site-packages/marionette_client-0.7.3-py2.7.egg/marionette/marionette_test.py", line 143, in run

testMethod()

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/system/test_privileged_app_contacts_prompt.py", line 28, in test_contact_prompt

permission.wait_for_permission_dialog_displayed()

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/apps/homescreen/regions/permission_dialog.py", line 17, in wait_for_permission_dialog_displayed

self.wait_for_element_displayed(*self._permission_dialog_locator)

File "/home/travis/build/mozilla-b2g/gaia/tests/python/gaia-ui-tests/gaiatest/apps/base.py", line 73, in wait_for_element_displayed

raise TimeoutException('Element %s present but not displayed before timeout' % locator)

TimeoutException: TimeoutException: Element permission-dialog present but not displayed before timeout

TEST-UNEXPECTED-FAIL | test_privileged_app_contacts_prompt.py test_privileged_app_contacts_prompt.TestPrivilegedAppContactsPrompt.test_contact_prompt |

----------------------------------------------------------------------

Ran 1 test in 22.006s

FAILED (errors=1)

Comment 1

[Kevin Grandon :kgrandon]

Zac - any idea what's going on here? The test failure is causing travis to go perma-red so I think we should close the tree for now..

Comment 2

[Zac C (:zac)]

Kevin, not in a position to debug this right now but this has regressed before which is why we have the test. Good chance it's legitmate. 
It should have been picked up on TBPL but we're not running the test there where the Gecko break might have been detected (apologies).

Leaving my ni? to investigate tomorrow morning but ni? also AndreiH in case he can get to it first.

Comment 3

[Kevin Grandon :kgrandon]

Just updating that the gaia tree is closed until we can determine what's causing this.

Comment 4

[Michael Henretty [:mikehenrty][:mhenretty]]

Looks like this is happening in TBPL too:
https://tbpl.mozilla.org/php/getParsedLog.php?id=33966885&tree=B2g-Inbound&full=1#error0

Comment 5

[Zac C (:zac)]

(In reply to Michael Henretty [:mhenretty] from comment #4)
> Looks like this is happening in TBPL too:
> https://tbpl.mozilla.org/php/getParsedLog.php?id=33966885&tree=B2g-
> Inbound&full=1#error0

No that's a different error.

Comment 6

[Michael Henretty [:mikehenrty][:mhenretty]]

(In reply to Zac C (:zac) from comment #5)
> (In reply to Michael Henretty [:mhenretty] from comment #4)
> > Looks like this is happening in TBPL too:
> > https://tbpl.mozilla.org/php/getParsedLog.php?id=33966885&tree=B2g-
> > Inbound&full=1#error0
> 
> No that's a different error.

Ah sorry, you're right. I wonder if they are related since one is the contact permission prompt and the other is adding a contact.

Comment 7

[Kevin Grandon :kgrandon]

I'm assuming that the cause of this test failure is bug 853356, but I'm currently trying to validate that. I may not have time to get to this tonight, so if someone could test it for me, I would appreciate it.

Comment 8

[Maire Reavy [:mreavy]]

See bug 948826 (which was added when bug 853356 bounced last time).  Is there a test case that bug 948826 didn't cover?

Comment 9

[Jason Smith [:jsmith]]

(In reply to Maire Reavy [:mreavy] from comment #8)
> See bug 948826 (which was added when bug 853356 bounced last time).  Is
> there a test case that bug 948826 didn't cover?

I don't think is a coverage problem - the test caught the failure here, so we need to find out an answer to this:

1. Can this be reproduced manually?

If yes, then we need to backout bug 855356. If no, then we need disable the test for now, fix it up, and re-enable with the fixes.

Comment 10

[Bob Silverberg [:bsilverberg]]

I haven't had any involvement with this test before, so I could be wrong, but looking at the code for the test, it does seem that I can manually reproduce it. If I open the UI tests - Privileged App, navigate to 'Contacts', and then tap the 'Insert fake contacts' button, I do not see a dialog asking me for permission, which the test seems to expect.

Comment 11

[Jason Smith [:jsmith]]

(In reply to Bob Silverberg [:bsilverberg] from comment #10)
> I haven't had any involvement with this test before, so I could be wrong,
> but looking at the code for the test, it does seem that I can manually
> reproduce it. If I open the UI tests - Privileged App, navigate to
> 'Contacts', and then tap the 'Insert fake contacts' button, I do not see a
> dialog asking me for permission, which the test seems to expect.

Okay - that implies that this patch broke the contacts API integration in a privileged app again. Let's get bug 853356 backed out again.

Comment 12

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #11)
> (In reply to Bob Silverberg [:bsilverberg] from comment #10)
> > I haven't had any involvement with this test before, so I could be wrong,
> > but looking at the code for the test, it does seem that I can manually
> > reproduce it. If I open the UI tests - Privileged App, navigate to
> > 'Contacts', and then tap the 'Insert fake contacts' button, I do not see a
> > dialog asking me for permission, which the test seems to expect.
> 
> Okay - that implies that this patch broke the contacts API integration in a
> privileged app again. Let's get bug 853356 backed out again.

This patch meaning the patch in bug 853356.

Comment 13

[Kevin Grandon :kgrandon]

Just landed a patch to disable this in gaia temporarily until we have a new b2g build: https://github.com/mozilla-b2g/gaia/commit/8b0e09bb13c3f7827ce8093ce969dca754414573

This is so we can get to green asap and re-open the tree.

Comment 14

[[:AndreiH]]

Thanks for investigating this guys!
This also fails on device.

Comment 15

[Maire Reavy [:mreavy]]

(In reply to Jason Smith [:jsmith] from comment #9)
> (In reply to Maire Reavy [:mreavy] from comment #8)
> > See bug 948826 (which was added when bug 853356 bounced last time).  Is
> > there a test case that bug 948826 didn't cover?
> 
> I don't think is a coverage problem - the test caught the failure here

I understand what you're saying, but I'm still somewhat confused.  We have a test to catch the failure, but the test is not on TBPL -- and that's why it looked like we were good to land bug 853356 when we weren't.  Why was bug 948826 resolved if the test hadn't been added to TBPL?  Even more important, can we get the test added to TBPL today?  Thanks.

Comment 16

[Zac C (:zac)]

This is my 'fault' if you must.

We were running the test on Travis CI. Generally we get tests running on there and then enable them on TBPL, for various reasons but it just took us some time to get to enabling it on TBPL.

Anyway you should never assume that a test is running just if the test suite is green. Tests are often aggressively disabled for a variety of reasons; the test may not be running.

I've already enabled the test on TBPL today.

Comment 17

[Maire Reavy [:mreavy]]

Hi Zac, I'm not looking for fault.  I'm sorry if it came across that way.  I'm looking to land bug 853356 (getUserMedia camera support) and have it stick finally. I've lost count, but I think this is the fifth or sixth time that bug 853356 has landed, looked like it was good/stable, and then was backed out a few days later.  

I was also genuinely wondering if I misunderstood the purpose of bug 948826: to add TBPL coverage.

(In reply to Zac C (:zac) from comment #16)
> Anyway you should never assume that a test is running just if the test suite is green. Tests are often aggressively disabled for a variety of reasons; the test may not be running.

I'm new to B2G and didn't realize tests were disabled that frequently.  What's the easiest way for someone to verify that this test is running?  Is it obvious what to look for?  


> I've already enabled the test on TBPL today.
Thanks!  We've just pushed a fresh TRY with the patch from bug 853356.

Comment 18

[Zac C (:zac)]

(In reply to Maire Reavy [:mreavy] from comment #17)
> 
> (In reply to Zac C (:zac) from comment #16)
> > Anyway you should never assume that a test is running just if the test suite is green. Tests are often aggressively disabled for a variety of reasons; the test may not be running.
> 
> I'm new to B2G and didn't realize tests were disabled that frequently. 
> What's the easiest way for someone to verify that this test is running?  Is
> it obvious what to look for?  
> 
When you click Gu in TBPL there should be a blob output which is a HTML report of the tests run, it should be listed in there. Similarly if it fails the debug information will be there.

We're also working on other ways to make the test state more transparent but they're "in the works" still.

> 
> > I've already enabled the test on TBPL today.
> Thanks!  We've just pushed a fresh TRY with the patch from bug 853356.

GL :)

Comment 19

[Zac C (:zac)]

Attachment: github pr

Comment 20

[Kevin Grandon :kgrandon]

Comment on attachment 8369446 [details] [review]
github pr

Thanks, and sorry for the typo!

Comment 21

[Kevin Grandon :kgrandon]

Landed: https://github.com/mozilla-b2g/gaia/commit/09297da87ea7147976f42599f43932f974146d64