928983 - Breaking out of a build on Windows with mach/mozmake often leaves zombie build processes

Status: NEW

(Firefox Build System :: General, defect)

Description

[Jim Mathies [:jimm]]

STR:

1) complete a full mach build
2) make a minor change to an internal src file that will trigger a relink of xul.dll
3) mach build binaries
4) during the final link of xul.dll, hit ctrl-c in the console
5) check the task manager

result: sometimes you see left over python, mozmake, or toolchain processes

Comment 1

[Gregory Szorc [:gps]]

This is possibly a mozprocess bug.

jhammel: Does this ring a bell?

Comment 2

[Gregory Szorc [:gps]]

We do run make with ignore_children=True. Source comment references bug 796840.

Comment 3

[Jeff Hammel]

This is probably the same as https://bugzilla.mozilla.org/show_bug.cgi?id=928324

Comment 4

[Mike Hommey [:glandium]]

(In reply to Jeff Hammel [:jhammel] from comment #3)
> This is probably the same as
> https://bugzilla.mozilla.org/show_bug.cgi?id=928324

I does look very similar.

Comment 5

[Jim Mathies [:jimm]]

In some (or all) cases these aren't zombie processes. I broke out during the middle of a build cycle (outside of linking xul.dll) and watched building continue in the task manager for a good minute or so. The console didn't display anything.

Comment 6

[Ben Turner (not reading bugmail, use the needinfo flag!)]

I have seen comment 5 also, and it causes subsequent rebuilds to fail if the other processes are still building behind the scenes.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Jonathan, this is a pretty annoying issue for Windows users (and there are other dupes on file for basically the same problem). Is there someone you might recommend who could look into sorting this out on the mozprocess side?

Comment 8

[Jonathan Griffin (:jgriffin)]

I have added this to the A-Team backlog in Trello (https://trello.com/b/3BjXQCEp/a-team-projects); will try to find an owner after this quarter's deliverables are done.

Comment 9

[(Away)]

I gave up hope a long time ago and just got in the habit of never doing ctrl+c on a build. I use this .cmd file as a workaround:

@taskkill /f /im mozmake.exe /t
@taskkill /f /im python.exe /t

Comment 10

[Julien Pagès (:parkouss)]

Can we add psutil as a mozprocess dependency ?

http://stackoverflow.com/a/4229404/924906

Comment 11

[Jonathan Griffin (:jgriffin)]

(In reply to Julien Pagès from comment #10)
> Can we add psutil as a mozprocess dependency ?
> 
> http://stackoverflow.com/a/4229404/924906

Yes, if we need to, although we'll need to check in a copy of it into the tree.  Does it solve this problem?

Comment 12

[Michael Shal [:mshal]]

(In reply to Jonathan Griffin (:jgriffin) from comment #11)
> (In reply to Julien Pagès from comment #10)
> > Can we add psutil as a mozprocess dependency ?
> > 
> > http://stackoverflow.com/a/4229404/924906
> 
> Yes, if we need to, although we'll need to check in a copy of it into the
> tree.  Does it solve this problem?

psutil is in-tree under python/psutil, though the one tricky part about it is that part of it needs to be compiled before it can be used (which happens during configure). As long as there is a fallback option if psutil fails to import, we can probably use that kill_proc_tree method.

Comment 13

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Still happening these days!
For me, it leaves some "cargo" tasks that never exit and that block running cargo in the next build. I need to manually end these tasks before I can restart the build.

Comment 15

[Mitchell Hentges [:mhentges] 🦀]

This may be solved by the move to MSYS2 and mintty.