Fix Firefox CI's dependence on TASKCLUSTER_WORKER_GROUP
Categories
(Taskcluster :: General, enhancement)
Tracking
(Not tracked)
People
(Reporter: bstack, Assigned: wcosta)
References
Details
Attachments
(5 files, 1 obsolete file)
Providers in w-m should be able to set workergroup to whatever they want. Currently we make an assumption that it maps to an AZ which can't necessarily always be true.
|
Reporter | |
Updated•5 years ago
|
|
||
Updated•5 years ago
|
|
|
||
Comment 1•5 years ago
|
||
My proposal here is to introduce $TASKCLUSTER_WORKER_LOCATION of the format <cloud>/<details>, available within tasks. The <details> portion is specific to the cloud. For example, it would be AZ for Amazon and zone for Google. Hardware workers (standalone and static) will be able to define this string arbitrarily.
Taskcluster-worker-runner would be responsible for supplying this to workers. Workers will fall back to unknown/unknown in a case where it is not supplied, so the env var is always set.
This will be documented in taskcluster-service docs along with other per-task variables, and with reference to tc-worker-runner for per-cloud definition of <details>.
This is simple, but presents a user-visible change, so should probably be handled with an RFC.
|
|
||
Comment 2•5 years ago
|
||
|
Assignee | |
Comment 4•5 years ago
|
||
|
|
Assignee | |
Comment 5•5 years ago
|
||
This version contains the TASKCLUSTER_WORKER_LOCATION environment
variable.
|
|
Reporter | |
Comment 6•5 years ago
|
||
|
||
Updated•5 years ago
|
|
|
Reporter | |
Comment 7•5 years ago
|
||
Create a worker pool to test builds in gcp
|
|
Assignee | |
Comment 8•5 years ago
|
||
|
|
Assignee | |
Comment 9•5 years ago
|
||
With the migration from AWS to GCP, we also need to migrate sccache
buckets from S3 to Google Storage.
The problem is how we deal with regions, since there isn't an exact
correspondence on the region names between the two cloud providers.
To make the transition smoother, docker-worker (and soon generic-worker)
provides a new environment variable called TASKCLUSTER_WORKER_LOCATION,
with information about the cloud provider the task is running on. Using
this new variable, we configure sccache to use the corresponding storage
service of the cloud provider where the task runs.
The bucket names in Google Storage are shorter because GCS imposes a
limit of 30 characteres for the names.
Ref: https://github.com/taskcluster/taskcluster-rfcs/pull/148/files
|
|
Assignee | |
Comment 10•5 years ago
|
||
|
||
Comment 11•5 years ago
|
||
Pushed by wcosta@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/63eed8246c06 Support sccache in Google Storage r=chmanchester,dustin
|
||
Comment 12•5 years ago
|
||
| bugherder | ||
|
|
Assignee | |
Comment 13•5 years ago
|
||
:grenade sccache support via gcpCredentials is ready, if you intend to implement in OCC, the only thing it is needed is the TASKCLUSTER_WORKER_LOCATION environment variable. It is a json string in the format:
{
"cloud": "google|aws"
"region": "<region name>"
"availabilityZone": "<zone>"
}
|
||
Comment 14•5 years ago
|
||
when bug 1580430 is resolved, we should be able to spin up windows gcp workers again and i will add this env var
|
|
||
Comment 15•5 years ago
|
||
this variable is now being set on gcp windows workers.
https://papertrailapp.com/events?q=system%3Agamma%20program%3AOpenCloudConfig%20TASKCLUSTER_WORKER_LOCATION
|
|
||
Comment 16•5 years ago
|
||
we have windows builds running in gcp again. these instances have their TASKCLUSTER_WORKER_LOCATION environment variable set as can be seen in papertrail or build logs. USE_SCCACHE is also set to 1 however these builds are not making successful use of sccache and build logs do not contain an ===SCCACHE STATS=== section.
Description
•