There should be a separate Taskcluster job that builds the tree with the tup backend. We'll need to have tup in tooltool for this.
I got pretty far with this, but the builds hit an issue to do with Docker: "fuse: failed to open /dev/fuse: Operation not permitted". From searching around about this, the workaround I can find suggests adding "--cap-add SYS_ADMIN --device /dev/fuse" to the `docker run` command line, which sounds like it will significantly compromise the container's isolation. I'm not sure if this is an option for TC or if there is a better workaround available.
Ok, I found the section in the docker docs: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities, which says we need "--cap-add SYS_ADMIN" for this. To add this would require patching docker-worker I expect, as in bug 1221661, but more to the point we were ok with adding that feature because it was only for testers, this could be a significant hurdle when we get to the point of evaluating a Tup build's suitability for shipping.
CAP_SYS_ADMIN is basically root and would almost completely undermine the process sandbox used by docker-worker. I reckon you'll have a hard time convincing people to give Docker containers that capability. See http://man7.org/linux/man-pages/man7/capabilities.7.html for more. I suspect there's a way to make fuse work without CAP_SYS_ADMIN. CAP_SYS_ADMIN is probably the most convenient. Although it likely requires changes to docker-worker to expose fuse to containers.
I bet the first problem here is that cgroups isn't allowing access to the /dev/fuse device. You can fix that as a one-off. Why it needs CAP_SYS_ADMIN, I dunno. Extended attributes?
mount() itself /might/ be safe to expose to the container because presumably the container was started with its own mount namespace. But, yeah, if you can't call mount() without CAP_SYS_ADMIN, then you are in a predicament. We may have to solve this by having the container started with a fuse filesystem pre-mounted. That requires docker-worker cooperation. That also means moving the tup fuse bits into docker-worker. That'll be a rabbit hole.
I'll also be looking at using an LD_PRELOAD shim again in tup instead of fuse, assuming that will work inside docker. If it does, that would get around us having to fiddle with capabilities I believe. (tup used to use LD_PRELOAD, but I switched it to fuse a while back for various reasons, like staticically-linked binary support).
LD_PRELOAD will work inside Docker as long as the library being loaded doesn't require system calls that the process doesn't have privileges to call :)