Open
Bug 1319518
Opened 7 years ago
Updated 2 years ago
Set sccache bucket from somewhere other than mozconfig.cache
Categories
(Firefox Build System :: General, defect)
Firefox Build System
General
Tracking
(Not tracked)
NEW
People
(Reporter: ted, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(1 file, 2 obsolete files)
mozconfig.cache is awfully complicated for a mozconfig: https://dxr.mozilla.org/mozilla-central/source/build/mozconfig.cache Most of the complexity is it trying to figure out which S3 bucket to use. I think we should just move this logic up a level and set the bucket name from mozharness, where we can do it in a couple of lines of Python. We already have the instance metadata available for buildbot builders, and taskcluster sets a $TASKCLUSTER_WORKER_GROUP environment variable with the AWS availability zone in it, and at the mozharness level we should know the branch we're building for, which is basically everything that we use to construct the bucket.
Updated•6 years ago
|
Assignee: nobody → mozilla
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 4•6 years ago
|
||
Please no. Not from mozharness. We should do less things in mozharness, not more. Decision task would probably be a better place for this.
Comment 5•6 years ago
|
||
At least the perfherder integration probably wants to stay in/move to mozharness. As mozharness currently handles downloading toolchains, I don't think starting sccache can move to something outside of that, currently. I'll investigate moving the logic of determining sccache configuration to the decision task, though.
Comment 6•6 years ago
|
||
(In reply to Tom Prince [:tomprince] from comment #5) > At least the perfherder integration probably wants to stay in/move to > mozharness. As mozharness currently handles downloading toolchains, I don't think this is entirely true. Toolchains from toolchain tasks are partly decision task and taskgraph transformations, and partly scripts (or mozharness) -- see |mach artifact toolchain| and associated environment variables. Toolchains from tooltool are not really tied to mozharness either, although mozharness will install a TOOLTOOL_MANIFEST if you ask it to do so. But see also tooltool_fetch throughout the taskcluster tasks, like at https://searchfox.org/mozilla-central/source/taskcluster/docker/centos6-build/system-setup.sh#285. Perhaps it's fair to say that mozharness currently handles toolchains in your tasks, but there's no reason to expand that set of responsibilities. I don't > think starting sccache can move to something outside of that, currently. > I'll investigate moving the logic of determining sccache configuration to > the decision task, though.
Updated•6 years ago
|
Attachment #8939112 -
Attachment is obsolete: true
Attachment #8939112 -
Flags: review?(core-build-config-reviews)
Updated•6 years ago
|
Attachment #8939113 -
Attachment is obsolete: true
Comment 7•6 years ago
|
||
mozconfig.cache and the way sccache integrates with the build system is horrible. It is bothering me as part of eliminating client.mk (bug 1412398). I was planning to move the sccache init/uninit into the `mach build` logic. I would prefer it not be in mozharness because we're actively trying to reduce the surface area of mozharness and move as much functionality as possible into `mach build`. (We want the task in CI to basically run `mach build`.)
Reporter | ||
Comment 8•6 years ago
|
||
I had a patch at one point that tried to start sccache from `mach build` but IIRC it failed in a way I never got around to figuring out on try and now I've lost the patch anyway.
Comment 9•6 years ago
|
||
Random thought: run-task might be the best place to handle things, because not everything we build use mach build or mozharness, but every one of them use run-task. Also kind of related: bug 1417421.
Comment 10•6 years ago
|
||
One wrinkle is that sccache wants a jobserver, and starting it anywhere but from make will result in sccache using a different job server that make, which might have issues.
Comment 11•6 years ago
|
||
Well, I don't really think we need to start sccache before any build system starts invoking a compiler. I was more thinking about all the setup that can't be done on the decision task, such as picking the right bucket (the decision task has, TTBOMK, no say in which AWS availability zone jobs are going to happen, so zone selection for the bucket needs to happen in the worker)
Reporter | ||
Comment 12•6 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #9) > Random thought: run-task might be the best place to handle things, because > not everything we build use mach build or mozharness, but every one of them > use run-task. This sounds pretty reasonable. One thought I had while pondering this was that we could perhaps split the resonsibility: the decision task could provide a list of environment variables containing template parameters to be filled in by run-task. Something like `SCCACHE_BUCKET=taskcluster-level-1-sccache-{region}`.
Comment 13•6 years ago
|
||
Updated•6 years ago
|
Product: Core → Firefox Build System
Comment 14•2 years ago
|
||
The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.
Assignee: mozilla → nobody
Updated•2 years ago
|
Severity: normal → S3
You need to log in
before you can comment on or make changes to this bug.
Description
•