Use TASKCLUSTER_ROOT_URL for all in-tree access to taskcluster

RESOLVED FIXED in Firefox -esr60

Status

task
RESOLVED FIXED
7 months ago
a month ago

People

(Reporter: dustin, Assigned: dustin)

Tracking

(Blocks 1 bug)

Trunk
mozilla66
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 fixed, firefox66 fixed)

Details

Attachments

(17 attachments, 5 obsolete attachments)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
..including the special-casing of https://taskcluster.net.
There are only 78 occurrences of `taskcluster.net` in the Gecko tree.  This might be a walk in the park!

I'll break fixing those down into microcommits.
Pete, many of these are cases where the code is making an externally-usable URL, while there are many cases (not showing up in this gist) that use the taskcluster proxy.  With respect to bug 1460015, the process sort of needs both rootUrls then: `http://taskcluster` (for the proxy) and `https://firefox-taskcluster.prod.mozaws.net` (for the externally-valid URLs).

What do you think about workers providing *two* environment variables to convey this information?  Maybe TASKCLUSTER_ROOT_URL (http://taskcluster) and TASKCLUSTER_PUBLIC_ROOT_URL (https://firefox-taskcluster.prod.mozaws.net)?

The other option is to only provide TASKCLUSTER_ROOT_URL=https://firefox-taskcluster.prod.mozaws.net and require tasks that use the proxy to override that value when using the proxy (in which case, do we provide the appropriate URL in another env var, or does everyone just hard-code http://taskcluster?)

I'm not sure which is best - what do you think?
Flags: needinfo?(pmoore)
Would the ROOT_URL be something that it'd be reasonable for the worker to provide? And, if that is the case, are there any cases where a task reasonably wants to use both the public and private URL? If not, the worker could provide one or the other, depending on whether the task has the proxy enabled. Even if so, it seems reasonable to provide both to the task. That way, the in-tree code can be used without change on multiple clusters (production and staging?).
> Would the ROOT_URL be something that it'd be reasonable for the worker to provide?

I think so.

> And, if that is the case, are there any cases where a task reasonably wants to use both the public and private URL?

Sure, the decision task both wants to create tasks (using the proxy) and include public URLs in descriptions, etc.

> If not, the worker could provide one or the other, depending on whether the task has the proxy enabled. Even if so, it seems reasonable to provide both to the task. That way, the in-tree code can be used without change on multiple clusters (production and staging?).

That's the ultimate goal, is that a task graph will easily run in either cluster.
(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #5)
> > And, if that is the case, are there any cases where a task reasonably wants to use both the public and private URL?
> 
> Sure, the decision task both wants to create tasks (using the proxy) and
> include public URLs in descriptions, etc.

My first reaction to this, is that we'd probably want to have the created tasks use the environment variables as much as possible. However, on reflection, there are probably lots of places where this isn't feasible, or sensible.
An example of a place that we can't use an env var is in the `mounts` property of generic-worker tasks -- those must be fully qualified URLs when the task is created.  Is that what you mean?
Oops, that's a terrible example, since generic-worker takes a taskId.  We could also change things to be able to take taskIds and paths, instead of installer URLs.

Per discussion with Pete, we're going to go with TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL.  The latter is only set when running in a task with a proxy, and must be used intentionally (and without credentials).

Pete is thinking about whether it makes sense to add support for this variable to clients, using the proxy by default if set.  But that doesn't block this particular project.
Flags: needinfo?(pmoore)
Mike, is the workaround landed in bug 1433033 to un-urlescape Location headers still required?  It's a bit trickier in a redeployability scenario since it's not clear what the redirect would be (we are probably not redeploying cloud-mirror, so it's even more than just a rootUrl problem).

If it is still an issue, is it practical to consider just re-escaping a portion of the path (regardless of the URL)?  Or, could we lean on upstream to fix the bug?  Thanks!
Flags: needinfo?(mh+mozilla)
and while I have you Mike, any idea what this error means?
  https://taskcluster-artifacts.net/TBSZsIM1Tju_2F8mCA6m_A/0/public/logs/live_backing.log
it looks like it couldn't reach snapshots over IPv6, which makes sense since EC2 doesn't support v6.  Any idea how my changeset would have led to that?
(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #13)
> and while I have you Mike, any idea what this error means?
>  
> https://taskcluster-artifacts.net/TBSZsIM1Tju_2F8mCA6m_A/0/public/logs/
> live_backing.log
> it looks like it couldn't reach snapshots over IPv6, which makes sense since
> EC2 doesn't support v6.  Any idea how my changeset would have led to that?

That happens sometimes. If EC2 doesn't support ipv6, we should disable ipv6 at the kernel level, then.

(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #11)
> Mike, is the workaround landed in bug 1433033 to un-urlescape Location
> headers still required?  It's a bit trickier in a redeployability scenario
> since it's not clear what the redirect would be (we are probably not
> redeploying cloud-mirror, so it's even more than just a rootUrl problem).
> 
> If it is still an issue, is it practical to consider just re-escaping a
> portion of the path (regardless of the URL)?  Or, could we lean on upstream
> to fix the bug?  Thanks!

I don't know. Try reverting the non-debian_package.py parts of 3bebaac29a1587e940136857dbf9fe6dae6ecffe and see what happens? That said, the workaround is a no-op if there's no redirection to cloud-mirror, so...
Flags: needinfo?(mh+mozilla)
(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #13)
> and while I have you Mike, any idea what this error means?
>  
> https://taskcluster-artifacts.net/TBSZsIM1Tju_2F8mCA6m_A/0/public/logs/
> live_backing.log
> it looks like it couldn't reach snapshots over IPv6, which makes sense since
> EC2 doesn't support v6.  Any idea how my changeset would have led to that?

https://phabricator.services.mozilla.com/D7365
Notes to self: this is ready for experimentation, but still failing with
  https://taskcluster-artifacts.net/ALFKJKt7Tou3lkgGO6vQdQ/0/public/logs/live_backing.log
I'm guessing because the existing image_builder image doesn't set TASKCLUSTER_ROOT_URL.
Depends on: 1498640
^^ rebased, linted, etc.  Fingers crossed!
In deb7-base:

[task 2018-12-10T22:37:49.158Z] executing ['sh', '-x', '-c', '   /builds/worker/checkouts/gecko/mach taskcluster-build-image   -t "debian7-base:90ba1640122af940143803aa3aac2f342617e3b720266d87da89d2907fc46dcf"   "debian7-base"']
[task 2018-12-10T22:37:49.160Z] + /builds/worker/checkouts/gecko/mach taskcluster-build-image -t debian7-base:90ba1640122af940143803aa3aac2f342617e3b720266d87da89d2907fc46dcf debian7-base
[task 2018-12-10T22:37:49.623Z] Step 0 : FROM debian:$DIST-$BASE_TAG
[task 2018-12-10T22:37:49.938Z] Pulling repository debian
[task 2018-12-10T22:38:15.785Z] Tag $DIST-$BASE_TAG not found in repository debian

also, the docker image hashes seem to be different in CI than when run manually..
`./mach vendor python taskcluster-urls==11.0.0`.
Eventually, workers will provide these variables directly
(https://bugzilla.mozilla.org/show_bug.cgi?id=1460015).  But for now, this
ensures that TASKCLUSTER_ROOT_URL is set everywhere, and TASKCLUSTER_PROXY_URL
is set wherever the proxy is active.

The setup for the mach commands defaults to https://taskcluster.net for user
convenience.  When the production instance's URL changes, we can simply change
that default.

This changes the docker build process to propagate TASKCLUSTER_ROOT_URL into
the docker images, and for good measure includes some code to use that value to
generate debian repo paths.
This provides an easy way to encode an artifact URL in static data such as
taskcluster/ci/nightly-l10n/kind.yml.  This could be used in
mozharness_test.py, for example, as well -- but other code (such as to support
backfilling) expects `task-reference` there.  To avoid breaking such subtle
bits, those can continue using `task-reference` with URLs generated based on
TASKCLUSTER_ROOT_URL.
This involves changing the calling signature (dropping some unused features)
and updating all three call sites to match. The function implementation is also
modified to use get_artifact_url.
Blocks: 1513732
Attachment #9030577 - Attachment is obsolete: true

Comment 51

4 months ago
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/85d7f8b330eb
vendor taskcluster-urls r=firefox-build-system-reviewers,gps
https://hg.mozilla.org/integration/autoland/rev/6dc9522ee0bf
set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL; r=tomprince,glandium
https://hg.mozilla.org/integration/autoland/rev/b93a0fcc86f3
add {artifact-reference: ..} r=tomprince
https://hg.mozilla.org/integration/autoland/rev/0c4101230d4d
use {artifact-reference: ..} in diffoscope; r=glandium
https://hg.mozilla.org/integration/autoland/rev/fb7cfca6ebc3
use {artifact-reference: ..} for symbol uploads; r=ted
https://hg.mozilla.org/integration/autoland/rev/5c92bb5ac895
Replace uses of `get_taskcluster_artifact_url` with `artifact-reference`; r=dustin
https://hg.mozilla.org/integration/autoland/rev/d3dfbd848236
use taskcluster-urls to create taskcluster URLs r=tomprince
https://hg.mozilla.org/integration/autoland/rev/c8b7e620eabf
use taskcluster_urls to generate interactive URL r=bstack
https://hg.mozilla.org/integration/autoland/rev/3017e5890739
move list_task_group to taskgraph.taskcluster.util r=bstack
https://hg.mozilla.org/integration/autoland/rev/e1ebad5d89c5
update libdmg-hfsplus to use TASKCLUSTER_ROOT_URL; r=glandium
https://hg.mozilla.org/integration/autoland/rev/5eb2c77d70a9
update fetch-content to use TASKCLUSTER_ROOT_URL; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/0b229b00818a
update diffoscope tasks to use TASKCLUSTER_ROOT_URL; r=glandium
https://hg.mozilla.org/integration/autoland/rev/90f3faa36137
update pipfile-updates to use TASKCLUSTER_ROOT_URL; r=tomprince
https://hg.mozilla.org/integration/autoland/rev/6d17291b8b92
update funsize scripts to use TASKCLUSTER_ROOT_URL; r=sfraser
https://hg.mozilla.org/integration/autoland/rev/f4945658d45f
update periodic-updates to use TASKCLUSTER_ROOT_URL; r=sfraser
https://hg.mozilla.org/integration/autoland/rev/31e500489665
generate portable URLs for Android mach commands; r=nalexander
Depends on: 1515137

Comment 53

4 months ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/3409b3f717ea
Port bug 1492664 - set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL. rs=bustage-fix
This is breaking developer artifact builds - they are failing to find the revisions because the root url isn't set:

======
 0:20.32 Attempting to find a pushhead containing 204ab379fb829827a82efd9ef72bb18acdcf42f6 on mozilla-central.
 0:21.10 Attempting to find a pushhead containing 204ab379fb829827a82efd9ef72bb18acdcf42f6 on integration/mozilla-inbound.
 0:21.96 Attempting to find a pushhead containing 204ab379fb829827a82efd9ef72bb18acdcf42f6 on releases/mozilla-beta.
 0:23.16 Retrieving the last 50 pushheads starting with id 110633 on integration/mozilla-inbound
 0:24.02 Retrieving the last 50 pushheads starting with id 35230 on mozilla-central
 0:24.77 Tried 70 pushheads, no built artifacts found.
 0:24.79 make[3]: *** [recurse_artifact] Error 1
======

If I do:

TASKCLUSTER_ROOT_URL=https://taskcluster.net ./mach build

then it works correctly.
Backed out 16 changesets (Bug 1492664) for breaking developer artifact builds, requested by standard8 a=backout

Backout push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel%2Crunnable&revision=28f92797e661694da8f10e3a9a8fbcfff49c919c
Status: RESOLVED → REOPENED
Flags: needinfo?(dustin)
Resolution: FIXED → ---
I suspect that backout is going to break things even worse :)
Flags: needinfo?(dustin)
There was an issue with the Ns jobs on the backout, but I don't see any more discussion in irc so I guess that's not a showstopper.  So there's no great rush to reland thus.
Duplicate of this bug: 1515188
https://hg.mozilla.org/integration/mozilla-inbound/rev/0699d3873e440453ce7d5cc7c398a1d076b92033
Bug 1492664 - vendor taskcluster-urls; r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/5ea6f03f845e49d503f5d0283557f54561c41654
Bug 1492664 - set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL; r=tomprince,glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/a972d6b4434edcbce1f93fb1629a6b81ca7bb585
Bug 1492664 - set TASKCLUSTER_ROOT_URL for mach build, too; r=froydnj

https://hg.mozilla.org/integration/mozilla-inbound/rev/9c35dd209c6b407bc3a45ce7b4c27272ef1bb486
Bug 1492664 - add {artifact-reference: ..}; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/c77ae59aba41a65f0e9d4896f9dfe3bf8a0e8353
Bug 1492664 - use {artifact-reference: ..} for symbol uploads; r=ted

https://hg.mozilla.org/integration/mozilla-inbound/rev/c566f1c8dcdfe6fdf73b0c0d8da66a14a7bfec08
Bug 1492664 - use {artifact-reference: ..} for l10n; r=aki

https://hg.mozilla.org/integration/mozilla-inbound/rev/335a92b1f42444dbcc4057f14bafab7f73f59e66
Bug 1492664 - use {artifact-reference: ..} in diffoscope; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/e7f30540870849cf6ba49957d37190fbd89b5f0d
Bug 1492664: Replace uses of `get_taskcluster_artifact_url` with `artifact-reference`; r=dustin

https://hg.mozilla.org/integration/mozilla-inbound/rev/29f33f22fd8b6998276e10c10aac2e36e64af4bb
Bug 1492664 - use taskcluster-urls to create taskcluster URLs; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/77699b51336b804336222192734922f76ea7ac10
Bug 1492664 - use taskcluster_urls to generate interactive URL; r=bstack

https://hg.mozilla.org/integration/mozilla-inbound/rev/870c8cec99e5f81814a8064a785c4c96f5f1aeca
Bug 1492664 - move list_task_group to taskgraph.taskcluster.util; r=bstack

https://hg.mozilla.org/integration/mozilla-inbound/rev/911b4b0fb6835fc7d0caaea5d64a590563ca3104
Bug 1492664 - update libdmg-hfsplus to use TASKCLUSTER_ROOT_URL; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/742b038bb1dd39c029afce73eb3f5b683cb590f2
Bug 1492664 - update fetch-content to use TASKCLUSTER_ROOT_URL; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/4c63a04fdd47c7cb8dc7350ae7fd6d861b525a2d
Bug 1492664 - update diffoscope tasks to use TASKCLUSTER_ROOT_URL; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/c1ea4a10cc8d3631d61c1172624373e3901debd7
Bug 1492664 - update pipfile-updates to use TASKCLUSTER_ROOT_URL; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/06c466ab432372eb645f07ca1ff215052492b358
Bug 1492664 - update funsize scripts to use TASKCLUSTER_ROOT_URL; r=sfraser

https://hg.mozilla.org/integration/mozilla-inbound/rev/d9a7f2ce49c3634baab1535315f9ba9f8f3b14ea
Bug 1492664 - update periodic-updates to use TASKCLUSTER_ROOT_URL; r=sfraser

https://hg.mozilla.org/integration/mozilla-inbound/rev/bf6d089640eb8c79b183129186cc936ac854a0bf
Bug 1492664 - generate portable URLs for Android mach commands; r=nalexander

https://hg.mozilla.org/integration/mozilla-inbound/rev/c82285d253de952fe6388e2ec5060057e2e8e80b
Bug 1492664 - update references in docs; r=froydnj

https://hg.mozilla.org/integration/mozilla-inbound/rev/2d876c4ece8b472b4c51d2240626974e96929a6e
Bug 1492664 - fix list_task_group to use GET; r=Callek
[Tracking Requested - why for this release]:
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 66

4 months ago
Backout by archaeopteryx@coole-files.de:
https://hg.mozilla.org/mozilla-central/rev/d0e13414d651
Backed out 21 changesets for breaking cron task for nightlies. a=backout
Flags: needinfo?(dustin)
Attachment #9032477 - Attachment is obsolete: true
Attachment #9032413 - Attachment is obsolete: true
Attachment #9032602 - Attachment is obsolete: true
https://hg.mozilla.org/integration/mozilla-inbound/rev/cc63b775369504c9299a88d9649e37f3f1d4c792
Bug 1492664 - vendor taskcluster-urls; r=gps

https://hg.mozilla.org/integration/mozilla-inbound/rev/bdaa57b4a2fdbc596f61ec77a42c4322ef9b48c1
Bug 1492664 - set TASKCLUSTER_ROOT_URL and TASKCLUSTER_PROXY_URL; r=tomprince,glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/5a7bea3fb23b1c208725be620aa881645af40f8a
Bug 1492664 - add {artifact-reference: ..}; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/edc2a5ed02cc62e0e6c5411e7160a4fd51415cbe
Bug 1492664 - use {artifact-reference: ..} for symbol uploads; r=ted

https://hg.mozilla.org/integration/mozilla-inbound/rev/aca20c673dd548d73399df4778faac77b722c44a
Bug 1492664 - use {artifact-reference: ..} for l10n; r=aki

https://hg.mozilla.org/integration/mozilla-inbound/rev/fdbc5b9746a2c417fbb3a27d96475e92dce7d2db
Bug 1492664 - use {artifact-reference: ..} in diffoscope; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/943f04bc1c706c5b1422bb746ef9acba7ecf285e
Bug 1492664: Replace uses of `get_taskcluster_artifact_url` with `artifact-reference`; r=dustin

https://hg.mozilla.org/integration/mozilla-inbound/rev/d3efbfd8e16250141fd58a9383678bbcf0ae6615
Bug 1492664 - use taskcluster-urls to create taskcluster URLs; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/1786aeb1e5c5227104c40dde4049f247071050d1
Bug 1492664 - use taskcluster_urls to generate interactive URL; r=bstack

https://hg.mozilla.org/integration/mozilla-inbound/rev/7433ce13608181f6e2738127c6fd0ab572109526
Bug 1492664 - move list_task_group to taskgraph.taskcluster.util; r=bstack

https://hg.mozilla.org/integration/mozilla-inbound/rev/aa78a567e81e3405b2eaf24ad2132f837c0f14fc
Bug 1492664 - update libdmg-hfsplus to use TASKCLUSTER_ROOT_URL; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/fbc8bc72b14ef0ca4b051b767f766535271edc44
Bug 1492664 - update fetch-content to use TASKCLUSTER_ROOT_URL; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/5f370da981050137d554e1d7d356a47606d158d0
Bug 1492664 - update diffoscope tasks to use TASKCLUSTER_ROOT_URL; r=glandium

https://hg.mozilla.org/integration/mozilla-inbound/rev/9bedcff5f5f4157e7f8c4e3cb97fc9d395bbaa0a
Bug 1492664 - update pipfile-updates to use TASKCLUSTER_ROOT_URL; r=tomprince

https://hg.mozilla.org/integration/mozilla-inbound/rev/f61ad7db75d493a04804eb9d797016a36d6e91a4
Bug 1492664 - update funsize scripts to use TASKCLUSTER_ROOT_URL; r=sfraser

https://hg.mozilla.org/integration/mozilla-inbound/rev/1f7b787349b838850fc4b73e29191efbf564d483
Bug 1492664 - update periodic-updates to use TASKCLUSTER_ROOT_URL; r=sfraser

https://hg.mozilla.org/integration/mozilla-inbound/rev/3808fda372b11df2e3db56c763b9364215f5fefe
Bug 1492664 - generate portable URLs for Android mach commands; r=nalexander

https://hg.mozilla.org/integration/mozilla-inbound/rev/9dfd14d495fecce69c0b500787a21c16d25322e9
Bug 1492664 - update references in docs; r=froydnj
(landed to inbound since autoland couldn't rebase but gave no useful output, but there were no conflicts on inbound)
Attachment #9032037 - Attachment is obsolete: true

Updated

4 months ago
See Also: → 1517342
You need to log in before you can comment on or make changes to this bug.