1114833 - [steeplechase] Mac leaves hanging diskimages-helper processes

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Nils Ohlmeier [:drno]]

All of the Mac's in the lab ran out of resources. It looks like they big amount of volumes mounted.

It looks to me like this line here:
https://github.com/mozilla/steeplechase/blob/master/steeplechase/runsteeplechase.py#L233
is mounting the Steeplechase volume twice. Running the umount command again makes sense, but mounting the steeplechase volume again after copying its content looks like copy and paste error to me.

Comment 1

[Nils Ohlmeier [:drno]]

Syd, does the code above look OK to you, or might this be a bug?

Comment 2

[(not currently active) Ted Mielczarek]

That code is just copying the .app package off of the DMG mount into a separate path. If you run a single test run on a mac from a fresh reboot, does the volume stay mounted? It's possible the unmount just isn't working properly for some reason.

Comment 3

[Nils Ohlmeier [:drno]]

Sorry I clicked in wrong line of code. This is the line I'm worried about:
https://github.com/mozilla/steeplechase/blob/master/steeplechase/runsteeplechase.py#L235

Comment 4

[Nils Ohlmeier [:drno]]

Also just realized that this line:
https://github.com/mozilla/steeplechase/blob/master/steeplechase/runsteeplechase.py#L237
doesn't have any logging.

Comment 5

[(not currently active) Ted Mielczarek]

I don't think that the exact line makes any difference there. :) line 233 is just assigning the command to the cmd variable, 234 logs it, and 235 executes it. The command in 237 is logged in the previous line, isn't it?

Comment 6

[Nils Ohlmeier [:drno]]

You are right Ted, I overlooked that it only logs what it is about to execute and not the result.
And I also overlooked that cmd gets redefined.

Never the less we have an issue on Mac. Yesterday I went into all the Mac's we have as part of our setup and manually killed all the hanging diskimage-helper processes. Since then the result now looks like this:

pf-mac10-10-01:~ mozilla$ ps auxwwww | grep diskim | wc -l
      47

The processes itself looks like this:

mozilla          9926   0.0  0.2  2503656   3656   ??  Ss    5:03PM   0:26.34 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid 42EC1FE2-4755-4DFC-A1CF-3B503C2B3B2F -post-exec 4

Comment 7

[(not currently active) Ted Mielczarek]

Are the disk images still mounted if you look at the output of `mount`, or is it just that these processes hang around?

Comment 8

[Nils Ohlmeier [:drno]]

No there is nothing mounted. But just these processes hanging around. But the problem with the hanging processes is that if enough of them hang around you can't even open a Terminal any more.

BTW you can only remove these hanging processes with SIGKILL, they don't die on SIGTERM.

I just watched one machine after manually cleaning the hanging processes before the test run:

During the test run these two processes get created:

pf-mac10-10-01:~ mozilla$ ps auxww | grep diskimage
mozilla         11912 100.8  0.6  2501568  13248   ??  R    11:05AM   0:08.56 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid EC7D4F92-1DBE-4F1D-B4B0-0D27159B348D -post-exec 4
mozilla         11916   0.0  0.0  2442588    436 s000  R+   11:06AM   0:00.00 grep diskimage
mozilla         11911   0.0  0.1  2442584   2368   ??  S    11:05AM   0:00.01 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid EC7D4F92-1DBE-4F1D-B4B0-0D27159B348D

Then the disk is mounted, stuff gets copied, disk gets unmounted, tests executes and at the end we still have this:

pf-mac10-10-01:~ mozilla$ ps auxww | grep diskimage
mozilla         11950   0.4  0.0  2432772    632 s000  S+   11:06AM   0:00.00 grep diskimage
mozilla         11912   0.0  0.8  2503488  17068   ??  Ss   11:05AM   0:25.64 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid EC7D4F92-1DBE-4F1D-B4B0-0D27159B348D -post-exec 4

Comment 9

[Nils Ohlmeier [:drno]]

Aha, apparently we gets lots of still attached disks:

http://stackoverflow.com/questions/4046019/error-when-detaching-volume-using-hdiutil-on-os-x

Comment 10

[(not currently active) Ted Mielczarek]

I was just about to post that same link, looks like we just need to switch "unmount" for "detach" to solve this.

Comment 11

[Nils Ohlmeier [:drno]]

Working on it...

Comment 12

[Nils Ohlmeier [:drno]]

I created a Pull request https://github.com/mozilla/steeplechase/pull/4
Ted can you review and merge please?

Comment 13

[Syd Polk :sydpolk]

I merged the pull request. hdiutil detach is the correct thing to do.

Comment 14

[Nils Ohlmeier [:drno]]

Verified. Works. No hanging processes any more.