921632 - [mozprocess] intermittent failure in test_process_kill_broad_wait (test_mozprocess_python.ProcTest)

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Jeff Hammel]

From https://bugzilla.mozilla.org/show_bug.cgi?id=778267#c31 ,
https://travis-ci.org/mozilla/mozbase/builds/11881700 :

======================================================================

FAIL: test_process_kill_broad_wait (test_mozprocess_python.ProcTest)

Process is started, we use a broad process tree, we let it spawn

----------------------------------------------------------------------

Traceback (most recent call last):

File
"/home/travis/build/mozilla/mozbase/mozprocess/tests/test_mozprocess_python.py",
line 201, in test_process_kill_broad_wait

p.didTimeout)

File
"/home/travis/build/mozilla/mozbase/mozprocess/tests/test_mozprocess_python.py",
line 271, in determine_status

self.assertTrue(not detected, "Detected process is still running,
process output: %s" % output)

AssertionError: Detected process is still running, process output:
travis 1067 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1068 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1069 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1071 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1072 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1073 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1074 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1075 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1076 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1078 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1079 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1081 1073 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1082 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1084 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1085 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1086 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1089 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1090 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1091 1082 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1092 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1097 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1098 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1099 1073 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1100 1092 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1102 1082 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1104 1092 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1109 1097 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1110 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1112 1103 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1116 1097 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1123 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1124 1097 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1125 1103 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1126 1115 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1127 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1132 1115 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1139 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1143 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1144 1127 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1149 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1151 1127 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1160 1127 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1164 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1168 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1179 1168 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1181 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1188 1168 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1189 1181 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1210 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1211 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1212 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1213 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1214 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1215 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1218 919 0 20:17 pts/2 00:00:00 grep
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py

----------------------------------------------------------------------

Ran 231 tests in 88.675s

FAILED (failures=1, skipped=2)

The command "python test.py" exited with 1.
"""

Rerunning the job gave a passing run.

From #ateam:

"""
13:30 < ffledgling> jhammel: just saw the failure
13:30 < jhammel> ffledgling: coolz; any clues?
13:30 < jhammel> ffledgling: guessing it is a genuine bug (blindly)
13:31 < ffledgling> jhammel: If I had to make a wild guess it would be
      that we're querying for the kill status before the OS can manage to
      shutdown all the processes/subprocesses
13:32 < ffledgling> looking into the output and googling for something
      more concrete though
13:33 < ffledgling> jhammel: is there someway I can repro the
      intermittent failures, with a couple of changes to the code?
13:34 < jhammel> ffledgling: heh, i can't faithfully say yes or no to
      that ;)
13:34 < jhammel> ffledgling: i don't know;  if you mean do we have
      record/playback...not so much
13:35 < ffledgling> jhammel:I guess I could setup travis with my own
      fork of the repo
13:36 < jhammel> ffledgling: yeah;  assuming its an environment
      factor, that might work
13:36 < jhammel> ffledgling: fwiw, i haven't seen locally
13:37 < ffledgling> jhammel: I haven't been able to repro locally
      either
13:37 < ffledgling> But I did get a similar-ish error on windows VM
13:37  * ffledgling tries the tests on a low powered VM
13:38 < jhammel> ffledgling: i'm guessing your guess is probably right
      (re querying kill status before shutdown)
"""

This should be fixed; I'm guessing it is a legitimate bug with
mozprocess.  If it can't quickly be fixed, these tests should probably
be disabled so as to not cause failures on m-c `make check`.

Comment 1

[Anhad Jai Singh (:ffledgling)]

Attachment: 0002-Bug-921632-mozprocess-intermittent-failure-restructu.patch

Attaching patch to split tests intogroups so that they are more easily managed by manifestdestiny, and disabling the failing intermittent test.

Also attaching travis-ci builds (relevant builds #18 to #22)
https://travis-ci.org/ffledgling/mozbase-testing-temp/builds

Side note: I got intermittent failures on my machine for the other tests too, if they crop up in the official travis, or worse on m-c/try we might have to disable them too.

Comment 2

[William Lachance (:wlach)]

Comment on attachment 812630 [details] [diff] [review]
0002-Bug-921632-mozprocess-intermittent-failure-restructu.patch

Review of attachment 812630 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good! I would prefer the name "proctest" to "utils" (which is rather vague).

Comment 3

[Anhad Jai Singh (:ffledgling)]

Attachment: 0002-Bug-921632-mozprocess-intermittent-failure-restructu.patch

mv utils.py -> proctest.py
refactored tests accordingly.
Passing build https://travis-ci.org/ffledgling/mozbase-testing-temp/builds/12012086

Comment 4

[William Lachance (:wlach)]

Comment on attachment 812661 [details] [diff] [review]
0002-Bug-921632-mozprocess-intermittent-failure-restructu.patch

Review of attachment 812661 [details] [diff] [review]:
-----------------------------------------------------------------

Good stuff, thanks (you didn't actually need to resubmit the patch, but no harm done!)

Comment 5

[Anhad Jai Singh (:ffledgling)]

(Forgot to mention.)
Pushed https://github.com/mozilla/mozbase/commit/6d2994acbdbce39d5b12a4426159329c3f90026f

Comment 6

[Jonathan Griffin (:jgriffin)]

Can this be closed?

Comment 7

[Jeff Hammel]

(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> Can this be closed?

The test is disabled but the issue still exists.  ABICT, it should stay open until it can be fixed.

Comment 8

[Geoff Brown [:gbrown]]

Mass closing bugs with no activity in 2+ years. If this bug is important to you, please re-open.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Not a wontfix. The failure didn't happen anymore because it is marked as skipped.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This intermittent failure existed a long time ago. I will have a look if it reproduces those days. If not lets just re-enable the test. Otherwise lets fix it before I work on other bugs to improve mozprocess.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 921632 - Re-enable delayed kill broad mozprocess unit test.

Review commit: https://reviewboard.mozilla.org/r/217842/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/217842/

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The try build was just fine, even we only run the tests on Linux64 those days:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=27b0eed90d5d89cf5a4c5a70bb22c215f808dd36&selectedJob=160407871

It also passes locally on MacOS.

Comment 13

[Andrew Halberstadt [:ahal]]

Comment on attachment 8948377 [details]
Bug 921632 - Re-enable delayed kill broad mozprocess unit test.

https://reviewboard.mozilla.org/r/217842/#review223624

Thanks for looking into this!

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8574698d5fb8
Re-enable delayed kill broad mozprocess unit test. r=ahal

Comment 15

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/8574698d5fb8

Comment 16

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/029925870825