1534647 - Perma mb TEST-UNEXPECTED-FAIL | No test output (missing mozunit.main() call?): /Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: aciure [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=233374281&repo=autoland

https://queue.taskcluster.net/v1/task/FJQr9bWMRmy04zK_RuIOuQ/runs/0/artifacts/public/logs/live_backing.log

[task 2019-03-12T14:08:40.716Z] 0:40.38 /Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozcrash/tests/test_symbols_path.py
[task 2019-03-12T14:08:40.716Z] 0:40.38 ============================= test session starts ==============================
[task 2019-03-12T14:08:40.716Z] 0:40.38 platform darwin -- Python 3.7.1, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /Users/cltbld/tasks/task_1552399473/checkouts/gecko/obj-x86_64-apple-darwin14.5.0/_virtualenvs/gecko-9o-SZLg7-/usr/local/bin/python3/bin/python
[task 2019-03-12T14:08:40.716Z] 0:40.38 rootdir: /Users/cltbld/tasks/task_1552399473/checkouts/gecko, inifile: /Users/cltbld/tasks/task_1552399473/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2019-03-12T14:08:40.717Z] 0:40.38 collecting ... collected 3 items
[task 2019-03-12T14:08:40.719Z] 0:40.38
[task 2019-03-12T14:08:40.720Z] 0:40.38 testing/mozbase/mozcrash/tests/test_symbols_path.py::test_symbols_path_not_present PASSED
[task 2019-03-12T14:08:40.720Z] 0:40.38 testing/mozbase/mozcrash/tests/test_symbols_path.py::test_symbols_path_unicode PASSED
[task 2019-03-12T14:08:40.720Z] 0:40.38 testing/mozbase/mozcrash/tests/test_symbols_path.py::test_symbols_path_url PASSED
[task 2019-03-12T14:08:40.720Z] 0:40.38
[task 2019-03-12T14:08:40.720Z] 0:40.38 =========================== 3 passed in 0.56 seconds ===========================
[task 2019-03-12T14:08:41.322Z] 0:40.98 /Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py
[task 2019-03-12T14:08:41.323Z] 0:40.98 Traceback (most recent call last):
[task 2019-03-12T14:08:41.323Z] 0:40.98 File "/Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py", line 12, in <module>
[task 2019-03-12T14:08:41.323Z] 0:40.98 from wptserve.handlers import handler
[task 2019-03-12T14:08:41.323Z] 0:40.98 File "/Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/web-platform/tests/tools/wptserve/wptserve/__init__.py", line 1, in <module>
[task 2019-03-12T14:08:41.323Z] 0:40.98 from .server import WebTestHttpd, WebTestServer, Router # noqa: F401
[task 2019-03-12T14:08:41.323Z] 0:40.98 File "/Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/web-platform/tests/tools/wptserve/wptserve/server.py", line 6, in <module>
[task 2019-03-12T14:08:41.326Z] 0:40.99 import ssl
[task 2019-03-12T14:08:41.326Z] 0:40.99 File "/tools/python37/lib/python3.7/ssl.py", line 98, in <module>
[task 2019-03-12T14:08:41.326Z] 0:40.99 import _ssl # if we can't import it, let the error propagate
[task 2019-03-12T14:08:41.326Z] 0:40.99 ModuleNotFoundError: No module named '_ssl'
[task 2019-03-12T14:08:41.326Z] 0:40.99 TEST-UNEXPECTED-FAIL | No test output (missing mozunit.main() call?): /Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py
[task 2019-03-12T14:08:41.327Z] 0:40.99 Setting retcode to 1 from /Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py
[task 2019-03-12T14:08:41.663Z] 0:41.32 /Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozfile/tests/test_extract.py
[task 2019-03-12T14:08:41.663Z] 0:41.32 ============================= test session starts ==============================
[task 2019-03-12T14:08:41.666Z] 0:41.33 platform darwin -- Python 3.7.1, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /Users/cltbld/tasks/task_1552399473/checkouts/gecko/obj-x86_64-apple-darwin14.5.0/_virtualenvs/gecko-9o-SZLg7-/usr/local/bin/python3/bin/python
[task 2019-03-12T14:08:41.666Z] 0:41.33 rootdir: /Users/cltbld/tasks/task_1552399473/checkouts/gecko, inifile: /Users/cltbld/tasks/task_1552399473/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2019-03-12T14:08:41.666Z] 0:41.33 collecting ... collected 4 items
[task 2019-03-12T14:08:41.666Z] 0:41.33
[task 2019-03-12T14:08:41.666Z] 0:41.33 testing/mozbase/mozfile/tests/test_extract.py::TestExtract::test_extract PASSED
[task 2019-03-12T14:08:41.666Z] 0:41.33 testing/mozbase/mozfile/tests/test_extract.py::TestExtract::test_extract_tarball PASSED
[task 2019-03-12T14:08:41.666Z] 0:41.33 testing/mozbase/mozfile/tests/test_extract.py::TestExtract::test_extract_zipfile PASSED
[task 2019-03-12T14:08:41.667Z] 0:41.33 testing/mozbase/mozfile/tests/test_extract.py::TestExtract::test_extract_zipfile_missing_file_attributes PASSED
[task 2019-03-12T14:08:41.667Z] 0:41.33
[task 2019-03-12T14:08:41.667Z] 0:41.33 =========================== 4 passed in 0.14 seconds ===========================
[task 2019-03-12T14:08:41.675Z] 0:41.33 /Users/cltbld/tasks/task_1552399473/checkouts/gecko/testing/mozbase/mozdevice/tests/test_is_app_installed.py
[task 2019-03-12T14:08:41.678Z] 0:41.34 ============================= test session starts ==============================
[task 2019-03-12T14:08:41.678Z] 0:41.34 platform darwin -- Python 3.7.1, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /Users/cltbld/tasks/task_1552399473/checkouts/gecko/obj-x86_64-apple-darwin14.5.0/_virtualenvs/gecko-9o-SZLg7-/usr/local/bin/python3/bin/python
[task 2019-03-12T14:08:41.678Z] 0:41.34 rootdir: /Users/cltbld/tasks/task_1552399473/checkouts/gecko, inifile: /Users/cltbld/tasks/task_1552399473/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2019-03-12T14:08:41.678Z] 0:41.34 collecting ... collected 5 items
[task 2019-03-12T14:08:41.678Z] 0:41.34
[task 2019-03-12T14:08:41.678Z] 0:41.34 testing/mozbase/mozdevice/tests/test_is_app_installed.py::test_is_app_installed PASSED
[task 2019-03-12T14:08:41.678Z] 0:41.34 testing/mozbase/mozdevice/tests/test_is_app_installed.py::test_is_app_installed_not_installed PASSED
[task 2019-03-12T14:08:41.679Z] 0:41.34 testing/mozbase/mozdevice/tests/test_is_app_installed.py::test_is_app_installed_partial_name PASSED
[task 2019-03-12T14:08:41.679Z] 0:41.34 testing/mozbase/mozdevice/tests/test_is_app_installed.py::test_is_app_installed_package_manager_error PASSED
[task 2019-03-12T14:08:41.679Z] 0:41.34 testing/mozbase/mozdevice/tests/test_is_app_installed.py::test_is_app_installed_no_installed_package_found PASSED

Comment 1

[Geoff Brown [:gbrown]]

This is a tier 2 perma-fail. It will eventually be duped to one of the see-also bugs, but let's leave it open for now to collect failure reports.

Comment 2

[Intermittent Failures Robot]

9 failures in 2771 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-beta: 6
• mozilla-inbound: 1

Platform breakdown:

• macosx64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1534647&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 3

[Intermittent Failures Robot]

4 failures in 2561 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 1

Platform breakdown:

• macosx64: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1534647&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #1)

It will eventually be duped to one of the see-also bugs

It turns out that's not true. This is a separate issue.

Comment 6

[Geoff Brown [:gbrown]]

There was some related discussion in bug 1529331, about python versions and updates. Of possible interest from that bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=1529331#c13

I am not aware of any python upgrades on the machines

:dividehex, do you know more about the python version on the osx machines?

Releng requested it in bug 1501497. Also relevant (and what spurred it): https://wiki.mozilla.org/ReleaseEngineering/Python_Standards#Updates

and https://bugzilla.mozilla.org/show_bug.cgi?id=1529331#c12

Not convinced it will work as it depends on a) Python 3.5 still being available on the macOS workers and b) Pipenv >> being able to locate Python 3.5 but I've pushed a patch to try that forces these tests to run on Python 3.5:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dfea6b543dde3d60bfe7882bb5daf66f3926565b

If Python 3.5 is indeed still available on the workers, then we may need to understand why Pipenv isn't
locating it. There have been new releases of Pipenv, and I believe it now uses pythonfinder to locate Python
on the system. Perhaps we could consider vendoring this, or implementing something similar ourselves.

My try push is failing on Windows due to "Python 3.5 was not found on your system…" and it looks like we're running
Python 3.6 for recent jobs on Windows. The macOS jobs are still pending, but the Linux jobs passed. We should pick
a Python version and make sure it's available across all of our platforms.

Comment 7

[Geoff Brown [:gbrown]]

I got a green run with python 3.6 hacked in:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=563bdd50b950049e8e841e84aa0e241764b24db3

Comment 8

[Geoff Brown [:gbrown]]

I think the python 3.7 installed via bug 1501497 must be lacking ssl support. Is it our own build? I think there are some extra steps to include ssl support...might that have been missed?

Comment 9

[Tarek Ziadé (:tarek)]

While having ssl support is an issue, I don't see any harm in skipping the test if there's no SSL support on the platform, because in these tests, it's just knowing if we can use wptserve.

Happy to provide the patch if you want, lmk with a needinfo

Comment 10

[Geoff Brown [:gbrown]]

Thanks Tarek, that's another good idea.

So we can resolve this several ways:

• fix python 3.7 on osx (comment 8)
• use python 3.6 (or maybe 3.5) on osx
• skip the test if ssl is not available

:davehunt, :catlee - Thoughts?

Comment 11

[Chris AtLee [:catlee]]

(In reply to Geoff Brown [:gbrown] from comment #8)

I think the python 3.7 installed via bug 1501497 must be lacking ssl support. Is it our own build? I think there are some extra steps to include ssl support...might that have been missed?

it's probably the case that the ssl development libraries weren't available on the machine where python was built. python's configure system silently disables important features like this if it can't find the right libraries locally.

I wonder if pyenv would help solve this?

Comment 12

[Intermittent Failures Robot]

28 failures in 3910 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-inbound: 4
• try: 3
• mozilla-central: 8

Platform breakdown:

• macosx64: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1534647&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

When we started to add Python 3 support to mozbase, it was decided to use Python 3.5 as our target version. I think it's great to support later versions, but still feel we should indicate the earliest version we support. The expectation was that the tests would run against this version, and the classifiers in the packages were updated accordingly. I think we should consider once more what our minimum supported version should be, ensure that version is available across all platforms, and run the tests using that version.

Comment 14

[Andrew Halberstadt [:ahal]]

Yes, I think it's a bug that the tests are running with 3.7 here in the first place. They should run with whatever version we intend to require to build Firefox (gps was aiming for 3.5, though not sure if this is still the case since he left).

Comment 15

[Geoff Brown [:gbrown]]

So is the releng update policy inappropriate? https://wiki.mozilla.org/ReleaseEngineering/Python_Standards#Updates

Or do we want the latest python3 to be available, in addition to older versions (or at least 3.5)?

These tests currently run /usr/local/bin/python3.

Comment 16

[Dave Hunt [:davehunt] [he/him] ⌚BST]

In bug 1529331 comment 12 I attempted to run the mozbase tests against Python 3.5, but it was not found on some platforms. I think it's fine to have later versions available, and to be explicit in the version we want to execute the tests against.

Comment 17

[Andrew Halberstadt [:ahal]]

Yeah, I don't think the policy is bad. We might even want to run some critically important tests against multiple versions of python 3. I think catlee's idea of using pyenv might be beneficial here. Maybe we can install pyenv + the python versions we care about in the docker images. Then tasks can do something like:

$ PYENV_VERSION=3.5 ./mach python-test ...

Comment 18

[Andrew Halberstadt [:ahal]]

Actually if the build tasks use this as well, there might not even be a need for that policy as it would all be controlled in-tree.

Edit: I guess OSX and Windows would still need out-of-tree updates, but we can still use pyenv there to make things easier.

Comment 19

[Intermittent Failures Robot]

18 failures in 461 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 10
• mozilla-central: 5
• mozilla-inbound: 2
• try: 1

Platform breakdown:

• macosx64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1534647&startday=2019-04-02&endday=2019-04-02&tree=all

Comment 20

[Andrew Halberstadt [:ahal]]

To be clear, this isn't the right bug to be figuring out how to handle multiple python versions. Let's just use gbrown's patch from comment 7 or skip this test on py3+osx for now.

Geoff, did you want to try and get your patch ready for review?

Comment 21

[Geoff Brown [:gbrown]]

Sure. I'm just waiting (and waiting...) for

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=95131caf480e468c9512d5614cc7f0c3ceda0ae9

Comment 22

[Geoff Brown [:gbrown]]

Attachment: Bug 1534647 - Switch python3 version to 3.6 on osx hosts; r=ahal

Quick fix for python3 mozbase perma-fail on osx: Use python 3.6 explicitly, rather
than the system default 3.7, which appears to be broken currently (lacking ssl support).

Comment 23

[Geoff Brown [:gbrown]]

Here's a quick fix.

I filed bug 1542327 to get osx python 3.7 fixed.

I'll leave it to others to sort out the general issue of handling multiple python versions.

Comment 24

[Intermittent Failures Robot]

37 failures in 2843 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 9
• autoland: 16
• mozilla-inbound: 10
• try: 2

Platform breakdown:

• macosx64: 37

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1534647&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f6bf51e87b1a
Switch python3 version to 3.6 on osx hosts; r=ahal

Comment 26

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/f6bf51e87b1a

Comment 27

[Intermittent Failures Robot]

9 failures in 3025 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 1
• mozilla-inbound: 3

Platform breakdown:

• macosx64: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1534647&startday=2019-04-08&endday=2019-04-14&tree=all