1596479 - hgweb private mirrors not receiving any traffic

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Connor Sheehan [:sheehan]]

I just noticed the private hgweb mirrors haven't been receiving any traffic for a few days. Scanning some logs, it looks like the change to TASKCLUSTER_WORKER_LOCATION environment variable was made and run-task was not updated accordingly.

I'll need to make some changes to run-task to take the new environment variable into account.

Comment 1

[Connor Sheehan [:sheehan]]

Attachment: Bug 1596479: teach `run-task` to get worker location from `TASKCLUSTER_WORKER_LOCATION` environment var r?tomprince

Previously we inspected the TASKCLUSTER_WORKER_GROUP environment variable,
which now only returns the cloud provider of the worker. This commit teaches
run-task to instead use the TASKCLUSTER_WORKER_LOCATION to gather
information about the location of the worker. We also use the extra data
about the cloud provider for the worker to construct a key for use in the
config, in the form cloudprovider/region, so GCP hgweb mirrors can be
amended to the hgmointernal config when they are ready.

While we're here we make the error handling for a missing environment
variable slightly nicer.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/efb782988830
teach `run-task` to get worker location from `TASKCLUSTER_WORKER_LOCATION` environment var r=tomprince

Comment 3

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/efb782988830

Comment 4

[Tom Prince [:tomprince]]

Based on the RFC, the only key that is guaranteed to be present is cloud, and that the rest are dependent on that value. I'm not sure how the various non gcp/aws workers are configured to report, currently.

Comment 5

[Cristian Brindusan [:cbrindusan]]

Backed out changeset efb782988830 (Bug 1596479) for wrench bustage (KeyError: 'region').

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=d65ded00cb51c7c5e3e1a23145b2a8bf15e5e52e&selectedJob=277254917

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=277254917&repo=autoland&lineNumber=32

Comment 6

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a1338baeaa0a
Backed out changeset efb782988830 for wrench bustage (KeyError: 'region'). CLOSED TREE

Comment 7

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/69a232cb6935
Backed out changeset efb782988830 for wrench bustage (KeyError: 'region'). CLOSED TREE

Comment 8

[Connor Sheehan [:sheehan]]

Thanks for the backout - it seems I misunderstood the JSON structure as Tom predicted in comment 4. I'll fix this up today.

Comment 9

[Tom Prince [:tomprince]]

(In reply to Connor Sheehan [:sheehan] from comment #8)

Thanks for the backout - it seems I misunderstood the JSON structure as Tom predicted in comment 4. I'll fix this up today.

Not so much as predicted, but diagnosed after seeing the failure and suggesting a backout.

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merged https://hg.mozilla.org/mozilla-central/rev/a1338baeaa0a

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5402371c61f7
teach `run-task` to get worker location from `TASKCLUSTER_WORKER_LOCATION` environment var r=tomprince

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/5402371c61f7