950866 - [Speedtests] Framework broken (for Firefox OS) after latest gaiatest update

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Robert Wood [:rwood]]

After the latest gaiatest update, the games benchmarking speedtests framework is broken. The following error occurs when attempting to run the tests on a device with FFOS:

python speedtests.py firefox -f conf/default -t octane-2.0 -t sunspider-1.0 -t kraken -t canvasmark --client="perfy-nexus4" --platform=ffos --cycles=30
/usr/local/lib/python2.7/dist-packages/mozrunner/utils.py:20: UserWarning: Module mozprocess was already imported from /home/speedtest/speedtests/client/mozprocess/__init__.pyc, but /usr/local/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
/usr/local/lib/python2.7/dist-packages/mozrunner/utils.py:20: UserWarning: Module mozlog was already imported from /home/speedtest/speedtests/client/mozlog/__init__.pyc, but /usr/local/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
/usr/local/lib/python2.7/dist-packages/mozrunner/utils.py:20: UserWarning: Module mozinfo was already imported from /home/speedtest/speedtests/client/mozinfo/__init__.pyc, but /usr/local/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
/usr/local/lib/python2.7/dist-packages/mozrunner/utils.py:20: UserWarning: Module mozdevice was already imported from /home/speedtest/speedtests/client/mozdevice/__init__.pyc, but /usr/local/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
Using port 46889
adbd is already running as root
Setting FirefoxOS Browser prefs...
20 KB/s (1671 bytes in 0.079s)
41 KB/s (1749 bytes in 0.041s)
Restarting b2g process...
Traceback (most recent call last):
  File "speedtests.py", line 380, in <module>
    main()
  File "speedtests.py", line 287, in main
    runner = BrowserRunner(evt, browsers, options.tests, testconfig)
  File "/home/speedtest/speedtests/client/BrowserRunner.py", line 236, in __init__
    self.browsers = BrowserRunner.browsers_by_os(platform)
  File "/home/speedtest/speedtests/client/BrowserRunner.py", line 161, in browsers_by_os
    return BrowserRunner.browsers_B2G()
  File "/home/speedtest/speedtests/client/BrowserRunner.py", line 142, in browsers_B2G
    B2GFirefoxBrowserController(os_name, 'firefox', package=None)
  File "/home/speedtest/speedtests/client/B2GBrowserControllers.py", line 186, in __init__
    super(B2GFirefoxBrowserController, self).__init__(os_name, browser_name, package, activity)
  File "/home/speedtest/speedtests/client/B2GBrowserControllers.py", line 43, in __init__
    self.dm.restart_b2g()
  File "/usr/local/lib/python2.7/dist-packages/gaiatest/gaia_test.py", line 720, in restart_b2g
    self.stop_b2g()
  File "/usr/local/lib/python2.7/dist-packages/gaiatest/gaia_test.py", line 750, in stop_b2g
    self.manager.shellCheckOutput(['stop', 'b2g'])
  File "/usr/local/lib/python2.7/dist-packages/gaiatest/gaia_test.py", line 666, in manager
    raise Exception('GaiaDevice has no device manager object set.')
Exception: GaiaDevice has no device manager object set.

Comment 1

[Robert Wood [:rwood]]

I believe it has to do with this change:

https://github.com/mozilla-b2g/gaia/pull/14476/files

mdas: Can you have a quick look please? Is there something I need to change on the speedtest framework side to work with the Mixin changes?

Comment 2

[Robert Wood [:rwood]]

:mdas, here's where it's failing in the speedtest framework:

https://github.com/Mozilla-Games/speedtests/blob/master/client/B2GBrowserControllers.py#L43

Comment 3

[Malini Das [:mdas] - Away, not checking bugmail]

GaiaDevice needs you to set its device_manager using add_device_manager, which is done in gaiatest by using marionette-client's B2GTestCaseMixin. Since this BrowserController object isn't a TestCase, it shouldn't use this mixin, but we should have an easier to access b2g manager. I've raised this concern over here https://bugzilla.mozilla.org/show_bug.cgi?id=852235#c9.

In the meantime, rwood agreed to set up the device_manager in this file without using the mixin.

Comment 4

[Robert Wood [:rwood]]

Thanks Malini, I took your suggestion and fixed it.

Another issue, :davehunt, this change:

https://github.com/mozilla-b2g/gaia/commit/76f5300fda681599d8ac8e91996ecc19c82225a6#diff-f8de369cfd0b2b0fb1029ce4ce18bd1d

Broke the speedtests framework here:

https://github.com/Mozilla-Games/speedtests/blob/master/client/B2GBrowserControllers.py#L174

I'm trying to fix it, but Dave can you have a look please? If I fix it first I will cancel the needinfo. Thanks!

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

You should be able to do:

def running(self):
    self.marionette.switch_to_frame()
    return 'browser' in gaiatest.GaiaApps(self.marionette).running_apps

You should make sure you're on the latest version of gaiatest though. The AppWindowManager changes landed on master but not v1.3 or v1.2 so you'll need the latest atoms to fallback to the previous window manager.

Comment 6

[Robert Wood [:rwood]]

Attachment: Link to pull request

Thanks Malini and Dave. 

Ack, here's the link to the pull request. Apologies for the breakage!

Comment 7

[Alan K [:ack]]

merged