Closed Bug 1319230 Opened 6 years ago Closed 5 years ago

Add a Taskcluster job to build with the tup backend


(Firefox Build System :: General, defect)

Not set


(firefox61 fixed)

Tracking Status
firefox61 --- fixed


(Reporter: mshal, Assigned: mshal)




(1 file)

There should be a separate Taskcluster job that builds the tree with the tup backend. We'll need to have tup in tooltool for this.
Assignee: nobody → cmanchester
I got pretty far with this, but the builds hit an issue to do with Docker: "fuse: failed to open /dev/fuse: Operation not permitted".

From searching around about this, the workaround I can find suggests adding "--cap-add SYS_ADMIN --device /dev/fuse" to the `docker run` command line, which sounds like it will significantly compromise the container's isolation. I'm not sure if this is an option for TC or if there is a better workaround available.
Ok, I found the section in the docker docs:, which says we need "--cap-add SYS_ADMIN" for this.

To add this would require patching docker-worker I expect, as in bug 1221661, but more to the point we were ok with adding that feature because it was only for testers, this could be a significant hurdle when we get to the point of evaluating a Tup build's suitability for shipping.
CAP_SYS_ADMIN is basically root and would almost completely undermine the process sandbox used by docker-worker. I reckon you'll have a hard time convincing people to give Docker containers that capability. See for more.

I suspect there's a way to make fuse work without CAP_SYS_ADMIN. CAP_SYS_ADMIN is probably the most convenient. Although it likely requires changes to docker-worker to expose fuse to containers.
I bet the first problem here is that cgroups isn't allowing access to the /dev/fuse device. You can fix that as a one-off. Why it needs CAP_SYS_ADMIN, I dunno. Extended attributes?
mount() itself /might/ be safe to expose to the container because presumably the container was started with its own mount namespace. But, yeah, if you can't call mount() without CAP_SYS_ADMIN, then you are in a predicament. We may have to solve this by having the container started with a fuse filesystem pre-mounted. That requires docker-worker cooperation. That also means moving the tup fuse bits into docker-worker. That'll be a rabbit hole.
I'll also be looking at using an LD_PRELOAD shim again in tup instead of fuse, assuming that will work inside docker. If it does, that would get around us having to fiddle with capabilities I believe. (tup used to use LD_PRELOAD, but I switched it to fuse a while back for various reasons, like staticically-linked binary support).
LD_PRELOAD will work inside Docker as long as the library being loaded doesn't require system calls that the process doesn't have privileges to call :)
Depends on: 1387098
Product: Core → Firefox Build System
Tup now has an LD_PRELOAD shim available again, and it appears to work fine in Docker. This is also nice because when we get to linking libxul, we won't have the same performance issues that we had with FUSE. I'll handle that in bug 1387098.
Depends on: 1449623
Assignee: cmanchester → mshal
Comment on attachment 8963393 [details]
Bug 1319230 - Add a Taskcluster job to build with the tup backend;

Looks sensible to me, but see my one small comment.  Exciting to see this moving forward!

::: testing/mozharness/configs/builds/releng_sub_linux_configs/
(Diff revision 1)
> +import os
> +
> +config = {

I expect you want something like here, since artifact builds probably don't make sense for Tup-backend builds.  (Yet?)
Attachment #8963393 - Flags: review+
Attachment #8963393 - Flags: review?(core-build-config-reviews)
(In reply to Nick Alexander :nalexander from comment #11)
> I expect you want something like
> builds/releng_sub_linux_configs/ here, since artifact
> builds probably don't make sense for Tup-backend builds.  (Yet?)

Ahh, good catch! I think chmanchester had artifact+tup working at one point, but it appears they don't work together now.
Attachment #8963393 - Flags: review?(core-build-config-reviews)
Pushed by
Add a Taskcluster job to build with the tup backend; r=nalexander
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
You need to log in before you can comment on or make changes to this bug.