All of the Mac's in the lab ran out of resources. It looks like they big amount of volumes mounted. It looks to me like this line here: https://github.com/mozilla/steeplechase/blob/master/steeplechase/runsteeplechase.py#L233 is mounting the Steeplechase volume twice. Running the umount command again makes sense, but mounting the steeplechase volume again after copying its content looks like copy and paste error to me.
Syd, does the code above look OK to you, or might this be a bug?
That code is just copying the .app package off of the DMG mount into a separate path. If you run a single test run on a mac from a fresh reboot, does the volume stay mounted? It's possible the unmount just isn't working properly for some reason.
Sorry I clicked in wrong line of code. This is the line I'm worried about: https://github.com/mozilla/steeplechase/blob/master/steeplechase/runsteeplechase.py#L235
Also just realized that this line: https://github.com/mozilla/steeplechase/blob/master/steeplechase/runsteeplechase.py#L237 doesn't have any logging.
I don't think that the exact line makes any difference there. :) line 233 is just assigning the command to the cmd variable, 234 logs it, and 235 executes it. The command in 237 is logged in the previous line, isn't it?
You are right Ted, I overlooked that it only logs what it is about to execute and not the result. And I also overlooked that cmd gets redefined. Never the less we have an issue on Mac. Yesterday I went into all the Mac's we have as part of our setup and manually killed all the hanging diskimage-helper processes. Since then the result now looks like this: pf-mac10-10-01:~ mozilla$ ps auxwwww | grep diskim | wc -l 47 The processes itself looks like this: mozilla 9926 0.0 0.2 2503656 3656 ?? Ss 5:03PM 0:26.34 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid 42EC1FE2-4755-4DFC-A1CF-3B503C2B3B2F -post-exec 4
Are the disk images still mounted if you look at the output of `mount`, or is it just that these processes hang around?
No there is nothing mounted. But just these processes hanging around. But the problem with the hanging processes is that if enough of them hang around you can't even open a Terminal any more. BTW you can only remove these hanging processes with SIGKILL, they don't die on SIGTERM. I just watched one machine after manually cleaning the hanging processes before the test run: During the test run these two processes get created: pf-mac10-10-01:~ mozilla$ ps auxww | grep diskimage mozilla 11912 100.8 0.6 2501568 13248 ?? R 11:05AM 0:08.56 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid EC7D4F92-1DBE-4F1D-B4B0-0D27159B348D -post-exec 4 mozilla 11916 0.0 0.0 2442588 436 s000 R+ 11:06AM 0:00.00 grep diskimage mozilla 11911 0.0 0.1 2442584 2368 ?? S 11:05AM 0:00.01 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid EC7D4F92-1DBE-4F1D-B4B0-0D27159B348D Then the disk is mounted, stuff gets copied, disk gets unmounted, tests executes and at the end we still have this: pf-mac10-10-01:~ mozilla$ ps auxww | grep diskimage mozilla 11950 0.4 0.0 2432772 632 s000 S+ 11:06AM 0:00.00 grep diskimage mozilla 11912 0.0 0.8 2503488 17068 ?? Ss 11:05AM 0:25.64 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid EC7D4F92-1DBE-4F1D-B4B0-0D27159B348D -post-exec 4
Aha, apparently we gets lots of still attached disks: http://stackoverflow.com/questions/4046019/error-when-detaching-volume-using-hdiutil-on-os-x
I was just about to post that same link, looks like we just need to switch "unmount" for "detach" to solve this.
Working on it...
I created a Pull request https://github.com/mozilla/steeplechase/pull/4 Ted can you review and merge please?
I merged the pull request. hdiutil detach is the correct thing to do.
Verified. Works. No hanging processes any more.