Skip-if filter for 'appname' not working if no mozinfo.json file is present

RESOLVED FIXED in Firefox 52

Status

defect
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: whimboo, Assigned: whimboo)

Tracking

Version 3
mozilla53
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox52 fixed, firefox53 fixed)

Details

Attachments

(1 attachment)

Marionette uses the skip-if filter for appname a lot, especially for skipping tests for Fennec. As I noticed today this filter is not working if you are connecting to an already running instance. 

https://dxr.mozilla.org/mozilla-central/rev/8103c612b79c2587ea4ca1b0a9f9f82db4b185b8/testing/marionette/harness/marionette/runner/base.py#933

By default appname is inserted via mozversion's `appname` property. But given that this doesn't exist in such a case we take the `app` argument. As such none of our tests which are marked to not run for Fennec are getting disabled.

manifest file:
--------------
[test_click_chrome.py]
skip-if = appname == 'fennec'
I can clearly reproduce this even locally with Firefox when using a filter like "appname == 'firefox'". I wonder why we correctly skip tests in automation?
Summary: Skip-if filter for 'appname' not working if connected without an instance → Skip-if filter for 'appname' not working
Comment on attachment 8819317 [details]
Bug 1323831 - Fix skip-if filter for appname if no mozinfo.json can be found.

https://reviewboard.mozilla.org/r/99138/#review99424
Attachment #8819317 - Flags: review?(ato) → review+
So we get the appname from somewhere else when we run the tests in automation:

> TypeError: active_tests() got multiple values for keyword argument 'appname'

Checking the code more I found the culprit, which lays here:

https://dxr.mozilla.org/mozilla-central/rev/63b447888a6469b9f6ae8f76ac5f0d7c6ea239da/testing/marionette/harness/marionette_harness/runner/base.py#937

We import more information from a json file into mozinfo. Here for the desktop builds:

mozinfo is: {u'bin_suffix': u'', u'pgo': False, u'sync': True, u'buildapp': u'browser', u'crashreporter': True, u'addon_signing': True, u'require_signing': False, u'release_or_beta': False, u'appname': u'firefox', u'stylo': False, u'mozconfig': u'/home/worker/workspace/build/src/.mozconfig', u'topsrcdir': u'/home/worker/workspace/build/src', u'telemetry': False, 'os_version': StringVersion ('16.04'), 'version': 'Ubuntu 16.04', u'datareporting': True, u'buildtype_guess': u'debug', 'bits': 64, 'has_sandbox': True, u'toolkit': u'gtk3', u'healthreport': True, u'asan': False, u'platform_guess': u'linux64', u'tests_enabled': True, 'linux_distro': 'Ubuntu', u'official': True, u'tsan': False, u'nightly_build': True, u'debug': True, 'os': u'linux', 'processor': u'x86_64'}

That means we have to pre-populate the list of values and pass all of them in once.
Btw. the mozinfo.json will only be present locally if you built Firefox/Fennec yourself and run it locally. Then this file will be present in obj/mozinfo.json.
Comment on attachment 8819317 [details]
Bug 1323831 - Fix skip-if filter for appname if no mozinfo.json can be found.

https://reviewboard.mozilla.org/r/99136/#review99468

Ok, the new commit fixed the issue. Not sure if Andreas is still around, so I will also request review from Maja.
Summary: Skip-if filter for 'appname' not working → Skip-if filter for 'appname' not working if no mozinfo.json file is present
Comment on attachment 8819317 [details]
Bug 1323831 - Fix skip-if filter for appname if no mozinfo.json can be found.

https://reviewboard.mozilla.org/r/99138/#review99470
Attachment #8819317 - Flags: review?(mjzffr) → review+
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/70acb25afe87
Fix skip-if filter for appname if no mozinfo.json can be found. r=ato,maja_zf
https://hg.mozilla.org/mozilla-central/rev/70acb25afe87
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
Test-only change which would be good to have for the next ESR release. Please uplift to aurora. Thanks.
Whiteboard: [checkin-needed-aurora]
Whiteboard: [checkin-needed-aurora]
You need to log in before you can comment on or make changes to this bug.