478150 - bm-xserve17 sometimes fails to kill processes

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

While looking at the mac leak test bustage on m-c I discovered this lot on bm-xserve17:

USER     PID %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
cltbld 65219  199.7  0.4   221696  16420   ??  R    Tue02PM 3458:34.12 /builds/buildbot/moz2-slave/mozilla-central-macosx-debug/build/obj-firefox/dist/MinefieldDebug.app/Contents/MacOS/firefox-bin -foreground -no-remote -profile /builds/buildbot/moz2-slave/mozilla-central-macosx-debug/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log

cltbld 89995   0.5  4.2   425652 175900   ??  S     4:44PM   5:53.30 /builds/buildbot/moz2-slave/mozilla-1.9.1-macosx-unittest/build/objdir/dist/Shiretoko.app/Contents/MacOS/firefox-bin -foreground -no-remote -profile /builds/buildbot/moz2-slave/mozilla-1.9.1-macosx-unittest/build/objdir/_tests/testing/mochitest/mochitesttestingprofile/ http://localhost:8888/tests/?autorun=1&closeWhenDone=1&consoleLevel=INFO

cltbld 94004   0.0  0.2   609840   8304   ??  S    28Jan09   0:51.71 /builds/buildbot/moz2-slave/mozilla-central-macosx-unittest/build/objdir/dist/bin/ssltunnel ../../../_profile/pgo/ssltunnel.cfg

The other prod slaves are fine, and I think I've had to clean up bm-xserve17 before, so there's something flaky about this buildbot slave. Perhaps a buildbot reinstall/upgrade would help - we have 0.7.7 on bm-xserveN and 0.7.9 on moz2-darwin9-slaveN.

Comment 1

[Chris AtLee [:catlee]]

And a few more:

cltbld 51686   0.0  0.2   612608   7888   ??  S    12:57AM   0:01.41 /builds/buildbot/moz2-slave/mozilla-central-macosx-debug/build/obj-firefox/dist/bin/ssltunnel /builds/buildbot/moz2-slave/mozilla-central-macosx-debug/build/obj-firefox/_leaktest/leakprofile/ssltunnel.cfg

cltbld 51687  296.5  1.6   340904  69100   ??  R    12:57AM 1055:08.72 /builds/buildbot/moz2-slave/mozilla-central-macosx-debug/build/obj-firefox/dist/MinefieldDebug.app/Contents/MacOS/firefox-bin -foreground -no-remote -profile /builds/buildbot/moz2-slave/mozilla-central-macosx-debug/build/obj-firefox/_leaktest/leakprofile/ http://localhost:8888/bloatcycle.html --trace-malloc malloc.log --shutdown-leaks=sdleak.log

Comment 2

[[e-mail] (:bhearsum)]

I did some preliminary investigation and haven't found any significant differences between bm-xserve17 and 18. Buildbot is exactly the same on both of them, twisted is the same version, as is python and zope-interface.

Comment 3

[alice nodelman [:alice] [:anode]]

Unless there's a fix in mind this should be moved to Future.

Comment 4

[alice nodelman [:alice] [:anode]]

Moving to future as a fix doesn't seem to be imminent.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

We should probably think about re-imaging this machine - had to clean up a hung leak test build.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

bm-xserve16 thru 19 will get re-imaged from the latest matser as part of bug 495336, so this bug should get fixed then.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Similar to bug 498393, this will be due to having "usepty = 0" in the slave's buildbot.tac. It'll still get the re-image, but the process killing bug should be fixed now.

Comment 8

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.