1265791 - [taskcluster-worker] [macosx-engine] Investigate how to isolate tasks inside Mac OSX engine

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect, P1)

Comment 1

[Pete Moore [:pmoore][:pete]]

Options I see:

1) chroot (https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man8/chroot.8.html)

Note, since we don't know what a task will want to do, it may be that we have to copy a significant portion of the OS into the chroot - this may be prohibitive, however, still worth investigating

2) running a VM and having the worker sit outside the VM

This appeals the most to me, since the task could potentially run as root, but not soil the worker environment. VMs can be discarded after task completion. This ties in with the work Jonas is doing atm.

3) Spawn low privilege users per task. This would be similar to how generic worker achieves the same on Windows. The idea is that the user can only affect the user environment, and this would be discarded after each task

4) Run as admin, but use some kind of deep-freeze/OS-restore to reimage machine after each task. This seems like a bad idea as tasks could potentially discover worker secrets and also potentially break the restore mechanism. But maybe these issues can be overcome.

5) Only run trusted tasks that are trusted to clean up after themselves, to keep a clean working environment. This seems like a bad idea, but would be akin to what buildbot currently does, AIUI.


My guess is number 2 will be the best long-term solution, and worth the investment. Pitch in if I missed any alternative solutions.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

For (1), you might consider bind-mounting things into the chroot, rather than copying.  Although googling suggests that bind mounting isn't well-supported in OS X.

(2) would be great, too, assuming it can be made reasonably efficient.  Even better if the construction of that image can be specified in-tree!  Again, mounting some shared directories into that VM may help (so maybe the VM disk image is just enough to boot, and most of /System, /build, and /tools are mounted from the host disk, and thus managed as filesystems rather than disk images?)

Comment 3

[Wander Lairson Costa]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> For (1), you might consider bind-mounting things into the chroot, rather
> than copying.  Although googling suggests that bind mounting isn't
> well-supported in OS X.
> 
> (2) would be great, too, assuming it can be made reasonably efficient.  Even
> better if the construction of that image can be specified in-tree!  Again,
> mounting some shared directories into that VM may help (so maybe the VM disk
> image is just enough to boot, and most of /System, /build, and /tools are
> mounted from the host disk, and thus managed as filesystems rather than disk
> images?)

The efficient VM approach requires processor support, and iirc, ec2 doesn't provide such support. I think we should go with approach 3 for the time being, and investigate options 1 and 3 as a follow up. We are going to have a meeting to talk about task isolation, then we can discuss this better.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> For (1), you might consider bind-mounting things into the chroot, rather
> than copying.  Although googling suggests that bind mounting isn't
> well-supported in OS X.

Chroots on OS X don't really work properly as I recall outside of POSIX calls.

Comment 5

[Wander Lairson Costa]

Approach we took in PR #77 was to create a new user on the fly and run the respective task with its privileges.