950401 - Missing process logging on Mac/BSD

Status: RESOLVED FIXED

(Core :: IPC, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: process-logging

Pretty self-explanatory. We need this code for zombie checking to work on tinderbox. I just copied the code from the Linux file.

Comment 1

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/55a2561bb201

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Backed out in http://hg.mozilla.org/integration/mozilla-inbound/rev/85c56bbcf37c for introducing several OSX mochitest timeouts like these: 
https://tbpl.mozilla.org/php/getParsedLog.php?id=32119410&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32119219&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=32120161&tree=Mozilla-Inbound


If this was expected and these are just existing timeouts that are now just being reported and need to be fixed, please say so and reland. :)

Comment 3

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I think these were existing timeouts, but I'm not able to work on this.

Comment 4

[(not currently active) Ted Mielczarek]

I pushed this to Try because we'll need this to fix bug 1143547 effectively and I don't believe this caused those timeouts:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c085c2c0812

Comment 5

[(not currently active) Ted Mielczarek]

Oops, silly build bustage from a header that has apparently been removed. Pushed again:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a7a1eba89579

Comment 6

[(not currently active) Ted Mielczarek]

Terrifyingly there seems to be a reproducible debug Jetpack crash on that Try push, so I rebased to a newer base changeset and tried again:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=881de5eb8b16

Comment 7

[(not currently active) Ted Mielczarek]

That Try push looks clean, I'll land this.

Comment 8

[(not currently active) Ted Mielczarek]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ad743e667004

Comment 9

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Backed out in:
https://hg.mozilla.org/integration/mozilla-inbound/rev/335f1295e99b
for causing miscellaneous orange on 10.6 browser-chrome mochitests, mostly on mochitest-e10s, but not entirely.

In particular, what I'm blaming on this are all the oranges involving:
FATAL ERROR: AsyncShutdown timeout in ShutdownLeaks: Wait for cleanup to be finished before checking for leaks
(which is a message from https://mxr.mozilla.org/mozilla-central/source/toolkit/components/asyncshutdown/AsyncShutdown.jsm that it writes shortly before calling gDebug.abort()), and sometimes preceded by some other failures.

Regression window narrowed with:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=Rev4%20MacOSX%20Snow%20Leopard%2010.6%20mozilla-inbound%20opt%20test%20mochitest-e10s-browser-chrome-3&fromchange=4b51391dc2a1&tochange=d084a35e8d79

I'll compile some failure logs in another comment.

Comment 10

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

https://treeherder.mozilla.org/logviewer.html#?job_id=8508779&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8490608&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8491954&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8493084&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8497716&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8508288&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8501714&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8505490&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8505491&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8505492&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8505747&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8507723&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8508465&repo=mozilla-inbound (on 10.10, not 10.6)
https://treeherder.mozilla.org/logviewer.html#?job_id=8508444&repo=mozilla-inbound (not mochitest-e10s, although I think some others above also were not)
https://treeherder.mozilla.org/logviewer.html#?job_id=8508722&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8508729&repo=mozilla-inbound
https://treeherder.mozilla.org/logviewer.html#?job_id=8508457&repo=mozilla-inbound

Comment 11

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

https://treeherder.mozilla.org/logviewer.html#?job_id=8508248&repo=mozilla-inbound (red for some reason)

Comment 12

[(not currently active) Ted Mielczarek]

Sorry about that, I got green on Try and didn't retrigger. I'll be more cautious with Try after I sort out why this is breaking things.

Comment 13

[Chris Manchester (limited bugmail, email directly)]

I pushed this to try and got ~50 green re-triggers of OSX 10.6 mochitest-e10s-browser-chrome [1]. If we can figure out a way to track child processes from outside the browser this wont be a problem, but if we can't I'm inclined to re-land this (and keep an eye on the trees).

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=532abb8ba1e1

Comment 14

[Chris Manchester (limited bugmail, email directly)]

I was able to get psutil installed on all the test slaves, I'm not intending to pursue this.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We need this feature for bug 1176758 to be able to kill Firefox via mozprocess after an update on OS X where it spawns itself into a new process group. With that we currently loose any control of the process. :(

So I will try another time to get this feature added. I actually had some code locally working before I found this bug, so there are some subtle differences to Bill's one attached here on that bug. I did a try push here:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ada87682e8ff

I will have a look later today how mochitest results look like on OS X, but if there are failures I cannot check them before over next week.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So far the try run looks promising. David and Ted, what specifically I would have to look out for? All of the links above are not usable anymore. I can't also find the mentioned mochitest(-browser) tests and assume those have been split up in multiple chunks meanwhile?

I will upload my patch because Bill missed to declare the global gProcessLog method.

Comment 17

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I really don't remember.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, and we also desupported OS X 10.6, so it might be a great time to try again! :)

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 950401 - Add process logging to OS X / BSD.

Review commit: https://reviewboard.mozilla.org/r/57648/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/57648/

Comment 20

[Benjamin Smedberg]

Comment on attachment 8759782 [details]
Bug 950401 - Add process logging to OS X / BSD.

https://reviewboard.mozilla.org/r/57648/#review55464

Comment 21

[(not currently active) Ted Mielczarek]

I never really looked into this very much, I was just trying to get the patch landed (as you are). I would say retrigger some extra Mochitest runs on OS X on try and make sure nothing looks broken, and then feel free to land it.

Per comment 13 this is probably fine anyway.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, try results look fine. I'm just going to push it. Lets fingers crossed that all will be fine now!

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/66211bb627bf
Add process logging to OS X / BSD. r=bsmedberg

Comment 24

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/66211bb627bf

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8759782 [details]
Bug 950401 - Add process logging to OS X / BSD.

Approval Request Comment
[Feature/regressing bug #]: New feature on OS X and BSD which is hidden behind an env variable
[User impact if declined]: None but necessary for test harnesses to track process ids of Firefox.
[Describe test coverage new/current, TreeHerder]: No changes
[Risks and why]: Will be only set by test harnesses or if the user explicitely enables it.
[String/UUID change made/needed]: None

Comment 26

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8759782 [details]
Bug 950401 - Add process logging to OS X / BSD.

Seems worth a try to uplift to improve testing capabilities on aurora. If we see any regressions from this please be ready to back it out.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/26ba0781a8e2