1053793 - Investigate if asserts in everything.me tests are reliable/correct

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect)

Description

[[:AndreiH]]

Recently we had Bug 1051888 and raised new questions.

Asserts like: https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py#L31

Is it expected for the app name to be in the app title/description?
I know tests were failing because of this.

Comment 1

[[:AndreiH]]

I will ask Viorela for more info on this, as I know she was the one investigating the past failures.
I cannot reproduce them now.

Comment 2

[Dylan Chun Wong [:dwong]]

Currently tests test_launch_everything_me_link and test_launch_everything_me_search are failing on the mozilla-central jobs.

http://jenkins1.qa.scl3.mozilla.com/job/flame.mozilla-central.ui.functional.non-smoke/87/HTML_Report/

http://jenkins1.qa.scl3.mozilla.com/job/flame.mozilla-central.ui.functional.smoke/71/HTML_Report/

The applications are running fine, and I can confirm >1 result is shown on the search, but the test asserts incorrectly it seems. I was able to reproduce the errors running the automated tests locally.

Comment 3

[Zac C (:zac)]

The test tries to distinguish between e.me links and marketplace links; possibly the result is not the kind of link we need to assert upon.

Comment 4

[Viorela Ioia [:viorela]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/23040

I modified the existing assert to check that the string we are searching for is in the link we tapped on - I noticed that each link has 'facebook' in its title. Please take a look and let me know if you agree with this change. 
Also, I ran the test several times with this patch and it passed.

Comment 5

[Zac C (:zac)]

Comment on attachment 8475228 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/23040

I think this is low risk to try, f+

Comment 6

[Florin Strugariu [:Bebe]]

Comment on attachment 8475228 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/23040

New assert looks OK

Comment 7

[Florin Strugariu [:Bebe]]

Is this ready for review?

Comment 8

[Viorela Ioia [:viorela]]

Yes, actually I believe we are ready to merge it, as I already got 2 f+

Comment 9

[Florin Strugariu [:Bebe]]

Comment on attachment 8475228 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/23040

This is failing on my end 


Might  be a different issue but I cant test this patch yet

TEST-UNEXPECTED-ERROR | test_everythingme_launch_link.py TestEverythingMeLaunchLink.test_launch_everything_me_link | JavascriptException: JavascriptException: NS_ERROR_UNEXPECTED: 
stacktrace:
	execute_async_script @gaia_test.py, line 50
	inline javascript, line 80
	src: "      mozPerms.set("
Traceback (most recent call last):
  File "/home/florinstrugariu/.virtualenvs/gaia/local/lib/python2.7/site-packages/marionette_client-0.8.2-py2.7.egg/marionette/marionette_test.py", line 153, in run
    self.setUp()
  File "/home/florinstrugariu/gaia/gaia/tests/python/gaia-ui-tests/gaiatest/tests/functional/everythingme/test_everythingme_launch_link.py", line 13, in setUp
    self.apps.set_permission('Search', 'geolocation', 'deny')
  File "/home/florinstrugariu/gaia/gaia/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 50, in set_permission
    (app_name, permission_name, value))
  File "/home/florinstrugariu/.virtualenvs/gaia/local/lib/python2.7/site-packages/marionette_client-0.8.2-py2.7.egg/marionette/marionette.py", line 1247, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/home/florinstrugariu/.virtualenvs/gaia/local/lib/python2.7/site-packages/marionette_client-0.8.2-py2.7.egg/marionette/decorators.py", line 35, in _
    return func(*args, **kwargs)
  File "/home/florinstrugariu/.virtualenvs/gaia/local/lib/python2.7/site-packages/marionette_client-0.8.2-py2.7.egg/marionette/marionette.py", line 634, in _send_message
    self._handle_error(response)
  File "/home/florinstrugariu/.virtualenvs/gaia/local/lib/python2.7/site-packages/marionette_client-0.8.2-py2.7.egg/marionette/marionette.py", line 682, in _handle_error
    raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)

TEST-INFO took 53167ms

SUMMARY
-------
passed: 0
failed: 1
todo: 0

Comment 10

[Florin Strugariu [:Bebe]]

Comment on attachment 8475228 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/23040

Let's wait for Bug 1064228 to get fixed and then we can review this pull again

Comment 11

[Viorela Ioia [:viorela]]

Attachment: Github PR: https://github.com/mozilla-b2g/gaia/pull/24081

Comment 12

[Florin Strugariu [:Bebe]]

https://github.com/mozilla-b2g/gaia/commit/5d0cb74c72e79fe188b01f344f4e3af320b83918