1529331 - Permafailing tier 2 testing/mozbase/mozprocess/tests/test_poll.py::ProcTestPoll::test_poll_after_kill_no_process_group TEST-UNEXPECTED-FAIL

Status: RESOLVED DUPLICATE of bug 1532630

(Testing :: Mozbase, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=229434209&repo=autoland

https://queue.taskcluster.net/v1/task/fvYKZuEUTZCTwMxWS48acQ/runs/0/artifacts/public/logs/live_backing.log

[task 2019-02-20T14:07:42.207Z] 0:52.54 testing/mozbase/mozprocess/tests/test_poll.py::ProcTestPoll::test_poll_after_kill_no_process_group TEST-UNEXPECTED-FAIL
[task 2019-02-20T14:07:42.207Z] 0:52.54 testing/mozbase/mozprocess/tests/test_poll.py::ProcTestPoll::test_poll_before_run PASSED
[task 2019-02-20T14:07:42.207Z] 0:52.54 testing/mozbase/mozprocess/tests/test_poll.py::ProcTestPoll::test_poll_while_running PASSED
[task 2019-02-20T14:07:42.207Z] 0:52.54
[task 2019-02-20T14:07:42.207Z] 0:52.54 =================================== FAILURES ===================================
[task 2019-02-20T14:07:42.207Z] 0:52.54 ______________ ProcTestPoll.test_poll_after_kill_no_process_group ______________
[task 2019-02-20T14:07:42.207Z] 0:52.54
[task 2019-02-20T14:07:42.207Z] 0:52.54 self = <test_poll.ProcTestPoll testMethod=test_poll_after_kill_no_process_group>
[task 2019-02-20T14:07:42.207Z] 0:52.54
[task 2019-02-20T14:07:42.207Z] 0:52.54 def test_poll_after_kill_no_process_group(self):
[task 2019-02-20T14:07:42.207Z] 0:52.54 """Process (no group) is killed, and poll() is called."""
[task 2019-02-20T14:07:42.207Z] 0:52.54 p = processhandler.ProcessHandler([self.python, self.proclaunch,
[task 2019-02-20T14:07:42.208Z] 0:52.54 "process_normal_finish_no_process_group.ini"],
[task 2019-02-20T14:07:42.208Z] 0:52.54 cwd=here,
[task 2019-02-20T14:07:42.208Z] 0:52.54 ignore_children=True
[task 2019-02-20T14:07:42.208Z] 0:52.54 )
[task 2019-02-20T14:07:42.208Z] 0:52.54 p.run()
[task 2019-02-20T14:07:42.208Z] 0:52.54 > returncode = p.kill()
[task 2019-02-20T14:07:42.208Z] 0:52.54
[task 2019-02-20T14:07:42.208Z] 0:52.54 testing/mozbase/mozprocess/tests/test_poll.py:70:
[task 2019-02-20T14:07:42.208Z] 0:52.54 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2019-02-20T14:07:42.208Z] 0:52.54 testing/mozbase/mozprocess/mozprocess/processhandler.py:812: in kill
[task 2019-02-20T14:07:42.208Z] 0:52.54 self.proc.kill(sig=sig)
[task 2019-02-20T14:07:42.208Z] 0:52.54 testing/mozbase/mozprocess/mozprocess/processhandler.py:208: in kill
[task 2019-02-20T14:07:42.208Z] 0:52.54 self.returncode = self.wait()
[task 2019-02-20T14:07:42.209Z] 0:52.54 testing/mozbase/mozprocess/mozprocess/processhandler.py:229: in wait
[task 2019-02-20T14:07:42.209Z] 0:52.54 self.returncode = self._wait()
[task 2019-02-20T14:07:42.209Z] 0:52.54 testing/mozbase/mozprocess/mozprocess/processhandler.py:665: in _wait
[task 2019-02-20T14:07:42.209Z] 0:52.54 subprocess.Popen.wait(self)
[task 2019-02-20T14:07:42.210Z] 0:52.55 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[task 2019-02-20T14:07:42.210Z] 0:52.55
[task 2019-02-20T14:07:42.211Z] 0:52.55 self = <mozprocess.processhandler.ProcessHandlerMixin.Process object at 0x10fad32e8>
[task 2019-02-20T14:07:42.211Z] 0:52.55 timeout = None
[task 2019-02-20T14:07:42.211Z] 0:52.55
[task 2019-02-20T14:07:42.211Z] 0:52.55 def wait(self, timeout=None):
[task 2019-02-20T14:07:42.211Z] 0:52.55 """Wait for child process to terminate; returns self.returncode."""
[task 2019-02-20T14:07:42.211Z] 0:52.55 if timeout is not None:
[task 2019-02-20T14:07:42.211Z] 0:52.55 endtime = _time() + timeout
[task 2019-02-20T14:07:42.212Z] 0:52.55 try:
[task 2019-02-20T14:07:42.212Z] 0:52.55 > return self._wait(timeout=timeout)
[task 2019-02-20T14:07:42.212Z] 0:52.55 E TypeError: _wait() got an unexpected keyword argument 'timeout'
[task 2019-02-20T14:07:42.212Z] 0:52.55
[task 2019-02-20T14:07:42.212Z] 0:52.55 /tools/python37/lib/python3.7/subprocess.py:984: TypeError

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Could be related to bug 1428713.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It appears that Python has been updated to 3.7.1 on macOS, which has caused this failure.

Comment 3

[Intermittent Failures Robot]

17 failures in 597 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 12
• mozilla-central: 5

Platform breakdown:

• macosx64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-02-21&endday=2019-02-21&tree=all

Comment 4

[Intermittent Failures Robot]

33 failures in 2602 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #42 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 9
• mozilla-central: 11
• mozilla-inbound: 13

Platform breakdown:

• macosx64: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-02-18&endday=2019-02-24&tree=all

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Dave, would you have the time to have a look into this failure?

Comment 6

[Bogdan Tara[:bogdan_tara | bogdant]]

Bug 1530738 seems to be a duplicate and is now perma failing.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=macosx64%2Copt%2Cpython%2C3%2Cunit%2Ctests%2Csource-test-python-mozbase-macosx64%2Fopt-py3%2Cpy3%28mb%29&tochange=2e02ffdb852e490a3467743fb91b2320721ff38c&fromchange=9f35adb8e466e30ccc7059bf5d787b8ab4054968

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Sorry, I do not. As mentioned in comment 7 the Python version on the macOS hardware has been updated to 3.7.1, which has introduced this failure. We either need to track down and revert that change, or we need to update moz(process|base) to support Pyhton 3.7.

Another option might be to ensure Python 3.5 is also available on our macOS hardware and restrict these tests to run on that version. I'm not sure who can answer regarding the Python versions on macOS, and the process for updating them.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel or Geoff, do you know who did that upgrade? Was it intentional? It feels strange given that we leave behind perma failures.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I am not aware of any python upgrades on the machines

:dividehex, do you know more about the python version on the osx machines?

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Not convinced it will work as it depends on a) Python 3.5 still being available on the macOS workers and b) Pipenv being able to locate Python 3.5 but I've pushed a patch to try that forces these tests to run on Python 3.5: https://treeherder.mozilla.org/#/jobs?repo=try&revision=dfea6b543dde3d60bfe7882bb5daf66f3926565b

If Python 3.5 is indeed still available on the workers, then we may need to understand why Pipenv isn't locating it. There have been new releases of Pipenv, and I believe it now uses pythonfinder to locate Python on the system. Perhaps we could consider vendoring this, or implementing something similar ourselves.

Comment 12

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #11)

Not convinced it will work as it depends on a) Python 3.5 still being available on the macOS workers and b) Pipenv being able to locate Python 3.5 but I've pushed a patch to try that forces these tests to run on Python 3.5: https://treeherder.mozilla.org/#/jobs?repo=try&revision=dfea6b543dde3d60bfe7882bb5daf66f3926565b

If Python 3.5 is indeed still available on the workers, then we may need to understand why Pipenv isn't locating it. There have been new releases of Pipenv, and I believe it now uses pythonfinder to locate Python on the system. Perhaps we could consider vendoring this, or implementing something similar ourselves.

My try push is failing on Windows due to "Python 3.5 was not found on your system…" and it looks like we're running Python 3.6 for recent jobs on Windows. The macOS jobs are still pending, but the Linux jobs passed. We should pick a Python version and make sure it's available across all of our platforms.

Comment 13

[Kendall Libby [:fubar] (he/him)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #10)

I am not aware of any python upgrades on the machines

:dividehex, do you know more about the python version on the osx machines?

Releng requested it in bug 1501497. Also relevant (and what spurred it): https://wiki.mozilla.org/ReleaseEngineering/Python_Standards#Updates

This feels to me like a disconnect between release engineering and testing/automation. /cc catlee

Comment 14

[Chris AtLee [:catlee]]

Yes, the intent is to always have up-to-date versions of python installed on the systems...but not at the expense of breaking things.

Non-virtualized hardware workers make this particularly challenging, as we have to support everything from trunk to ESR branches in the same environment.

Would one option be to make python3.5 available in parallel with python3.7?

Comment 15

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Chris AtLee [:catlee] from comment #14)

Would one option be to make python3.5 available in parallel with python3.7?

Yes, that should work and Pipenv is meant to be able to locate it. I think it looks for a python3.5 binary in the path. My try push in comment 12 sets the desired Python version to 3.5 but it's failing on Win10 and macOS because it's not found.

Comment 16

[Intermittent Failures Robot]

29 failures in 2981 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 15
• mozilla-inbound: 3

Platform breakdown:

• macosx64: 29

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 19

[Tarek Ziadé (:tarek)]

I added Bug 1532630 without knowing this one existed. Sorry!

I have a patch that should work on all py version by accepting the timeout option if provided.

Comment 20

[Intermittent Failures Robot]

18 failures in 592 pushes (0.03 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18

Platform breakdown:

• macosx64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-03-06&endday=2019-03-06&tree=all

Comment 21

[Intermittent Failures Robot]

52 failures in 2818 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #35 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 15
• mozilla-inbound: 7
• autoland: 30

Platform breakdown:

• macosx64: 52

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-03-04&endday=2019-03-10&tree=all

Comment 22

[Bogdan Tara[:bogdan_tara | bogdant]]

In the last 7 days there have been 57 occurrences on Mac OSX 64 opt.

Comment 23

[Intermittent Failures Robot]

47 failures in 2771 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #40 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-release: 1
• mozilla-beta: 11
• autoland: 13
• mozilla-central: 10
• mozilla-inbound: 12

Platform breakdown:

• macosx64: 47

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 24

[Intermittent Failures Robot]

18 failures in 512 pushes (0.035 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 6
• mozilla-inbound: 3
• autoland: 9

Platform breakdown:

• macosx64: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-03-19&endday=2019-03-19&tree=all

Comment 25

[Stefan Hindli [:stefan_hindli]]

Over the last 7 days there have been 37 occurence on macosx64.

Here is the most recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235635968&repo=mozilla-inbound&lineNumber=383

Comment 26

[Intermittent Failures Robot]

38 failures in 2561 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 1
• autoland: 17
• mozilla-inbound: 7
• mozilla-central: 13

Platform breakdown:

• macosx64: 38

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-03-18&endday=2019-03-24&tree=all

Comment 27

[Geoff Brown [:gbrown]]

(In reply to Tarek Ziadé (:tarek) from comment #19)

I added Bug 1532630 without knowing this one existed. Sorry!

I have a patch that should work on all py version by accepting the timeout option if provided.

:tarek - Are you going to go ahead with your solution in bug 1532630? (I think it will be okay if you just correct the seconds/milliseconds issue.)

Comment 28

[Tarek Ziadé (:tarek)]

Yes, sorry I was busy on other stuff, will try to finish it this week

Comment 29

[Intermittent Failures Robot]

20 failures in 557 pushes (0.036 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 11
• mozilla-inbound: 3
• autoland: 6

Platform breakdown:

• macosx64: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-03-26&endday=2019-03-26&tree=all

Comment 30

[Tarek Ziadé (:tarek)]

NOte that https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235635968&repo=mozilla-inbound&lineNumber=383 is yet another failure on the _ssl modulde. This is unrelated to my mozprocess fix

Comment 31

[Geoff Brown [:gbrown]]

Yes, test_poll_after_kill_no_process_group is fine now - thanks!

The ssl failure continues: bug 1534647.

Comment 32

[Intermittent Failures Robot]

34 failures in 3910 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 8
• mozilla-central: 19
• mozilla-inbound: 7

Platform breakdown:

• macosx64: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-03-25&endday=2019-03-31&tree=all

Comment 33

[Intermittent Failures Robot]

2 failures in 2843 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-inbound: 1

Platform breakdown:

• macosx64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-04-01&endday=2019-04-07&tree=all

Comment 34

[Intermittent Failures Robot]

1 failures in 3025 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• macosx64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-04-08&endday=2019-04-14&tree=all

Comment 35

[Intermittent Failures Robot]

3 failures in 4658 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 3

Platform breakdown:

• macosx64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1529331&startday=2019-05-13&endday=2019-05-19&tree=all

Comment 36

[Bob Clary [:bc] (inactive)]

These 3 are all misclassifications

task 2019-05-13T09:50:48.759Z]  0:41.29 /Users/cltbld/tasks/task_1557740802/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py
[task 2019-05-13T09:50:48.759Z]  0:41.29 Traceback (most recent call last):
[task 2019-05-13T09:50:48.759Z]  0:41.29   File "/Users/cltbld/tasks/task_1557740802/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py", line 12, in <module>
[task 2019-05-13T09:50:48.759Z]  0:41.29     from wptserve.handlers import handler
[task 2019-05-13T09:50:48.760Z]  0:41.29   File "/Users/cltbld/tasks/task_1557740802/checkouts/gecko/testing/web-platform/tests/tools/wptserve/wptserve/__init__.py", line 1, in <module>
[task 2019-05-13T09:50:48.762Z]  0:41.29     from .server import WebTestHttpd, WebTestServer, Router  # noqa: F401
[task 2019-05-13T09:50:48.762Z]  0:41.29   File "/Users/cltbld/tasks/task_1557740802/checkouts/gecko/testing/web-platform/tests/tools/wptserve/wptserve/server.py", line 6, in <module>
[task 2019-05-13T09:50:48.762Z]  0:41.29     import ssl
[task 2019-05-13T09:50:48.763Z]  0:41.29   File "/tools/python37/lib/python3.7/ssl.py", line 98, in <module>
[task 2019-05-13T09:50:48.763Z]  0:41.29     import _ssl             # if we can't import it, let the error propagate
[task 2019-05-13T09:50:48.763Z]  0:41.29 ModuleNotFoundError: No module named '_ssl'
[task 2019-05-13T09:50:48.763Z]  0:41.29 TEST-UNEXPECTED-FAIL | No test output (missing mozunit.main() call?): /Users/cltbld/tasks/task_1557740802/checkouts/gecko/testing/mozbase/mozfile/tests/test_load.py

Filed Bug 1552777.

Issues with the logging: ModuleNotFoundError not being used in suggestions, time stamp part of bug filer's suggestion for summary which warns if you attempt to remove it.