Closed Bug 921632 Opened 11 years ago Closed 6 years ago

[mozprocess] intermittent failure in test_process_kill_broad_wait (test_mozprocess_python.ProcTest)

Categories

(Testing :: Mozbase, defect)

Product:
Testing
Component:
Mozbase
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr52 wontfix, firefox58 wontfix, firefox59 fixed, firefox60 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla60
Tracking Flags:
Tracking Status
firefox-esr52 --- wontfix
firefox58 --- wontfix
firefox59 --- fixed
firefox60 --- fixed

People

(Reporter: k0scist, Assigned: whimboo)

References

(Blocks 1 open bug)

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 1 obsolete file)

0002-Bug-921632-mozprocess-intermittent-failure-restructu.patch
25.31 KB, patch
wlach
: review+
Details | Diff | Splinter Review
Bug 921632 - Re-enable delayed kill broad mozprocess unit test.
59 bytes, text/x-review-board-request
ahal
: review+
Details 
From https://bugzilla.mozilla.org/show_bug.cgi?id=778267#c31 ,
https://travis-ci.org/mozilla/mozbase/builds/11881700 :

======================================================================

FAIL: test_process_kill_broad_wait (test_mozprocess_python.ProcTest)

Process is started, we use a broad process tree, we let it spawn

----------------------------------------------------------------------

Traceback (most recent call last):

File
"/home/travis/build/mozilla/mozbase/mozprocess/tests/test_mozprocess_python.py",
line 201, in test_process_kill_broad_wait

p.didTimeout)

File
"/home/travis/build/mozilla/mozbase/mozprocess/tests/test_mozprocess_python.py",
line 271, in determine_status

self.assertTrue(not detected, "Detected process is still running,
process output: %s" % output)

AssertionError: Detected process is still running, process output:
travis 1067 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1068 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1069 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1071 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1072 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1073 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1074 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1075 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1076 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1078 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1079 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1081 1073 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1082 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1084 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1085 1067 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1086 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1089 1068 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1090 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1091 1082 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1092 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1097 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1098 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1099 1073 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1100 1092 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1102 1082 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1104 1092 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1109 1097 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1110 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1112 1103 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1116 1097 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1123 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1124 1097 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1125 1103 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1126 1115 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1127 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1132 1115 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1139 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1143 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1144 1127 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1149 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1151 1127 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1160 1127 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1164 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1168 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1179 1168 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1181 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1188 1168 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1189 1181 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1210 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1211 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1212 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1213 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1214 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1215 1 0 20:17 pts/2 00:00:00
/home/travis/virtualenv/python2.7/bin/python
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py
process_normal_broad_python.ini

travis 1218 919 0 20:17 pts/2 00:00:00 grep
/home/travis/build/mozilla/mozbase/mozprocess/tests/proclaunch.py

----------------------------------------------------------------------

Ran 231 tests in 88.675s

FAILED (failures=1, skipped=2)

The command "python test.py" exited with 1.
"""

Rerunning the job gave a passing run.

From #ateam:

"""
13:30 < ffledgling> jhammel: just saw the failure
13:30 < jhammel> ffledgling: coolz; any clues?
13:30 < jhammel> ffledgling: guessing it is a genuine bug (blindly)
13:31 < ffledgling> jhammel: If I had to make a wild guess it would be
      that we're querying for the kill status before the OS can manage to
      shutdown all the processes/subprocesses
13:32 < ffledgling> looking into the output and googling for something
      more concrete though
13:33 < ffledgling> jhammel: is there someway I can repro the
      intermittent failures, with a couple of changes to the code?
13:34 < jhammel> ffledgling: heh, i can't faithfully say yes or no to
      that ;)
13:34 < jhammel> ffledgling: i don't know;  if you mean do we have
      record/playback...not so much
13:35 < ffledgling> jhammel:I guess I could setup travis with my own
      fork of the repo
13:36 < jhammel> ffledgling: yeah;  assuming its an environment
      factor, that might work
13:36 < jhammel> ffledgling: fwiw, i haven't seen locally
13:37 < ffledgling> jhammel: I haven't been able to repro locally
      either
13:37 < ffledgling> But I did get a similar-ish error on windows VM
13:37  * ffledgling tries the tests on a low powered VM
13:38 < jhammel> ffledgling: i'm guessing your guess is probably right
      (re querying kill status before shutdown)
"""

This should be fixed; I'm guessing it is a legitimate bug with
mozprocess.  If it can't quickly be fixed, these tests should probably
be disabled so as to not cause failures on m-c `make check`.
Blocks: 917750
See Also: → 778267
Attaching patch to split tests intogroups so that they are more easily managed by manifestdestiny, and disabling the failing intermittent test.

Also attaching travis-ci builds (relevant builds #18 to #22)
https://travis-ci.org/ffledgling/mozbase-testing-temp/builds

Side note: I got intermittent failures on my machine for the other tests too, if they crop up in the official travis, or worse on m-c/try we might have to disable them too.
Attachment #812630 - Flags: review?(wlachance)
Attachment #812630 - Flags: feedback?(jhammel)
Comment on attachment 812630 [details] [diff] [review]
0002-Bug-921632-mozprocess-intermittent-failure-restructu.patch

Review of attachment 812630 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good! I would prefer the name "proctest" to "utils" (which is rather vague).
Attachment #812630 - Flags: review?(wlachance) → review+
mv utils.py -> proctest.py
refactored tests accordingly.
Passing build https://travis-ci.org/ffledgling/mozbase-testing-temp/builds/12012086
Attachment #812630 - Attachment is obsolete: true
Attachment #812630 - Flags: feedback?(jhammel)
Attachment #812661 - Flags: review?(wlachance)
Comment on attachment 812661 [details] [diff] [review]
0002-Bug-921632-mozprocess-intermittent-failure-restructu.patch

Review of attachment 812661 [details] [diff] [review]:
-----------------------------------------------------------------

Good stuff, thanks (you didn't actually need to resubmit the patch, but no harm done!)
Attachment #812661 - Flags: review?(wlachance) → review+
Can this be closed?
(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> Can this be closed?

The test is disabled but the issue still exists.  ABICT, it should stay open until it can be fixed.
Mass closing bugs with no activity in 2+ years. If this bug is important to you, please re-open.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Not a wontfix. The failure didn't happen anymore because it is marked as skipped.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
This intermittent failure existed a long time ago. I will have a look if it reproduces those days. If not lets just re-enable the test. Otherwise lets fix it before I work on other bugs to improve mozprocess.
status-firefox58: --- → wontfix
status-firefox59: --- → affected
status-firefox60: --- → affected
status-firefox-esr52: --- → affected
Keywords: intermittent-failure, test-disabled
Attachment #8948377 - Flags: review?(ahalberstadt)
The try build was just fine, even we only run the tests on Linux64 those days:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=27b0eed90d5d89cf5a4c5a70bb22c215f808dd36&selectedJob=160407871

It also passes locally on MacOS.
Assignee: nobody → hskupin
Blocks: 1434878
Status: REOPENED → ASSIGNED
Comment on attachment 8948377 [details]
Bug 921632 - Re-enable delayed kill broad mozprocess unit test.

https://reviewboard.mozilla.org/r/217842/#review223624

Thanks for looking into this!
Attachment #8948377 - Flags: review?(ahalberstadt) → review+
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8574698d5fb8
Re-enable delayed kill broad mozprocess unit test. r=ahal
https://hg.mozilla.org/mozilla-central/rev/8574698d5fb8
Status: ASSIGNED → RESOLVED
Closed: 6 years ago → 6 years ago
status-firefox60: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: