1033033 - gaiatest test_endurance_camera_photo.py fails at session_capability attribute

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::UI Tests, defect, P1)

Description

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

Attachment: mariinette fails for test_endurance_camera_photo.py

test_endurance_camera_photo.py fails with following logs :


  if self.device_manager:
  File "/net/cros-bs-p4/local/mnt/workspace/tkundu/kk_36/out/host/marionette/client/marionette/marionette_auto_venv/lib/python2.7/site-packages/marionette_client-0.7.10-py2.7.egg/marionette/runner/mixins/b2g.py", line 50, in device_manager
    return self.get_device_manager()
  File "/net/cros-bs-p4/local/mnt/workspace/tkundu/kk_36/out/host/marionette/client/marionette/marionette_auto_venv/lib/python2.7/site-packages/marionette_client-0.7.10-py2.7.egg/marionette/runner/mixins/b2g.py", line 43, in get_device_manager
    capabilities = self.marionette.session_capabilities
AttributeError: 'NoneType' object has no attribute 'session_capabilities'
(marionette_auto_venv)tkundu@tkundu-linux:/net/cros-bs-p4/local/mnt/workspace/tkundu/kk_36$ 
(marionette_auto_venv)tkundu@tkundu-linux:/net/cros-bs-p4/local/mnt/workspace/tkundu/kk_36$ 


I found that we are using marionette_client-0.7.10-py2.7.egg package and this python package contains patch from bug 1019043

If we use same marionette_client-0.7.10-py2.7.egg but revert bug 1019043 from that python package (marionette_client-0.7.10-py2.7.egg) then issue goes away.


Issue is not visible for all gaiatest . 
But some gaiatest like test_endurance_camera_photo.py gives above error.
This is a regression .

Comment 1

[Jason Smith [:jsmith]]

Clearing the blocking flag - we will not block on test only changes.

Comment 2

[Michael Vines [:m1] [:evilmachines]]

Jason, this needs to be fixed ASAP as we are blocked with our testing as a result of all this mozrunner churn.  If we can't test then we can't make the release clearly.

Comment 3

[Michael Vines [:m1] [:evilmachines]]

Tapas, I think you have a workaround for this issue.  Could you please attach that here, so we can look at it and discuss if/how to fix properly.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The traceback in comment 0 is incomplete. It appears that at the time the device_manager property is accessed there is no Marionette object. I have replicated this and include the full trace below:

Traceback (most recent call last):
  File "/Users/dhunt/.virtualenvs/gaia/bin/gaiatest", line 9, in <module>
    load_entry_point('gaiatest==0.25', 'console_scripts', 'gaiatest')()
  File "/Users/dhunt/workspace/gaia/tests/python/gaia-ui-tests/gaiatest/runtests.py", line 60, in main
    cli(runner_class=GaiaTestRunner, parser_class=GaiaTestOptions)
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runtests.py", line 28, in cli
    runner = startTestRunner(runner_class, options, tests)
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runtests.py", line 19, in startTestRunner
    runner.run_tests(tests)
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runner/base.py", line 823, in run_tests
    self.run_test_sets()
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runner/base.py", line 1054, in run_test_sets
    self.run_test_set(in_process_tests)
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runner/base.py", line 1031, in run_test_set
    self.run_test(test['filepath'], test['expected'], test['oop'])
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runner/base.py", line 1000, in run_test
    **self.test_kwargs)
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/marionette_test.py", line 336, in add_tests_to_suite
    **kwargs))
  File "/Users/dhunt/workspace/gaia/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 915, in __init__
    MemoryEnduranceTestCaseMixin.__init__(self, *args, **kwargs)
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runner/mixins/endurance.py", line 136, in __init__
    if self.device_manager:
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runner/mixins/b2g.py", line 50, in device_manager
    return self.get_device_manager()
  File "/Users/dhunt/workspace/mozilla/mozilla-central/testing/marionette/client/marionette/runner/mixins/b2g.py", line 43, in get_device_manager
    capabilities = self.marionette.session_capabilities
AttributeError: 'NoneType' object has no attribute 'session_capabilities'

It looks like the endurance tests were previously relying on the existence of a device manager to suggest we're running against a physical device. Since bug 1019043 we use the Marionette session's capabilities to determine if we're running on a desktop build. Previously, we would have to try/catch the device_manager call. This means that a Marionette session is required when getting the device manager, and one is not available in the initialisation of MemoryEnduranceTestCaseMixin.

Removing the if statement from http://hg.mozilla.org/mozilla-central/file/default/testing/marionette/client/marionette/runner/mixins/endurance.py#l136 appears to fix this for me, however we will then likely see exceptions if these are run against desktop builds. Alternatively, we will need to delay the initialisation of MemoryEnduranceTestCaseMixin until after a Marionette session has been started.

Comment 5

[cmtalbert]

Looks like Dave has your work around there, as long as you don't need to run against desktop builds. We've been making a lot of changes to this code base to update it, and it could be that these tests just need to be updated. Some of these tests may not have been updated since there were a few weeks where we considered retiring them in favor of the AreWeSlimYet tests. In the end we decided to keep the endurance tests, but we likely missed updates as a result of that prioritization change. NeedInfo'ing Jgriffin to take a look.

Comment 6

[Jonathan Griffin (:jgriffin)]

Attachment: Don't access the device_manager during __init__,

This is the simplest and lowest-impact fix that will cause execution to be the same.

Comment 7

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 8449696 [details] [diff] [review]
Don't access the device_manager during __init__,

Review of attachment 8449696 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, thanks

Comment 8

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/73a651b7e30a

Depending on how you're running the endurance tests, we may need to bump and release a new marionette-client for this.

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/73a651b7e30a

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/4c30346e553b

Comment 11

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> Created attachment 8449696 [details] [diff] [review]
> Don't access the device_manager during __init__,
> 
> This is the simplest and lowest-impact fix that will cause execution to be
> the same.

Thanks for fixing this issue in gecko.

But I am still seeing this issue is present in [1][2] 

[1] https://github.com/mozilla-b2g/gaia/blob/v2.0/tests/python/gaia-ui-tests/requirements.txt
[2] https://pypi.python.org/pypi/marionette_client/0.7.10
[3] Please check this file in 7.10: marionette_client-0.7.10-py2.7.egg/marionette/runner/mixins
/endurance.py

Do we need to change requirments.txt[1] manually and upgrade to different python packages ?

Comment 12

[Zac C (:zac)]

Yes, you need 0.7.11.

I'll update the branches to use this version later today.

Comment 13

[Zac C (:zac)]

Sorry 0.7.11 is still not out yet

Comment 14

[Tapas[:tkundu on #b2g/gaia/memshrink/gfx] (always NI me)]

(In reply to Zac C (:zac) from comment #13)
> Sorry 0.7.11 is still not out yet

I think that we should keep it opened till we see 0.7.11 is released. Our automation tool is using 0.7.10 [1] [2] [3] and reproducing this bug always.


[1] https://github.com/mozilla-b2g/gaia/blob/v2.0/tests/python/gaia-ui-tests/requirements.txt
[2] https://pypi.python.org/pypi/marionette_client/0.7.10
[3] Please check this file in 7.10: marionette_client-0.7.10-py2.7.egg/marionette/runner/mixins
/endurance.py

Comment 15

[Jason Smith [:jsmith]]

(In reply to Tapas Kumar Kundu from comment #14)
> (In reply to Zac C (:zac) from comment #13)
> > Sorry 0.7.11 is still not out yet
> 
> I think that we should keep it opened till we see 0.7.11 is released. Our
> automation tool is using 0.7.10 [1] [2] [3] and reproducing this bug always.

That isn't how we track bugs in bugzilla. When a patch lands in the respective branches, then the bug is considered fixed. A release of a product is tracked separately.

Comment 17

[Zac C (:zac)]

(In reply to Tapas Kumar Kundu from comment #14)
> (In reply to Zac C (:zac) from comment #13)
> > Sorry 0.7.11 is still not out yet
> 
> I think that we should keep it opened till we see 0.7.11 is released. Our
> automation tool is using 0.7.10 [1] [2] [3] and reproducing this bug always.
> 
> 
> [1]
> https://github.com/mozilla-b2g/gaia/blob/v2.0/tests/python/gaia-ui-tests/
> requirements.txt
> [2] https://pypi.python.org/pypi/marionette_client/0.7.10
> [3] Please check this file in 7.10:
> marionette_client-0.7.10-py2.7.egg/marionette/runner/mixins
> /endurance.py

You need to wait until 0.7.11 is relased.

Comment 18

[Michael Vines [:m1] [:evilmachines]]

(In reply to Zac C (:zac) from comment #17)
> (In reply to Tapas Kumar Kundu from comment #14)
> > (In reply to Zac C (:zac) from comment #13)
> > > Sorry 0.7.11 is still not out yet
> > 
> 
> You need to wait until 0.7.11 is relased.

How do we track this?

Comment 19

[Zac C (:zac)]

It is marked as a blocking bug of this one.

Comment 20

[Brogan Zumwalt [Inactive]]

STR involves automation processes that are unusable to create test case.