1543893 - TASKCLUSTER_ROOT_URL is not set on terraform-packet

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Mike Hommey [:glandium]]

I got failures on Android x86 tests that run on packet.net when I added fetches:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=239839533&repo=try&lineNumber=31

[taskcluster 2019-04-12 03:49:44.837Z] Worker ID: machine-8
[taskcluster 2019-04-12 03:49:44.837Z] Worker Group: packet-sjc1
[taskcluster 2019-04-12 03:49:44.837Z] Worker Node Type: packet.net
[taskcluster 2019-04-12 03:49:44.837Z] Worker Type: gecko-t-linux
[taskcluster 2019-04-12 03:49:44.837Z] Public IP: 147.75.109.15
[taskcluster 2019-04-12 03:49:44.837Z] using cache "gecko-level-1-checkouts-v3-33ea6ead87f10b63cd64" -> /builds/worker/checkouts
[taskcluster 2019-04-12 03:49:44.837Z] using cache "gecko-level-1-tooltool-cache-v3-33ea6ead87f10b63cd64" -> /builds/worker/tooltool-cache
[taskcluster 2019-04-12 03:49:46.610Z] Image 'public/image.tar.zst' from task 'XixQ3m-oS3C-95iK9P4Gmg' loaded.  Using image ID sha256:6a0e1b6ec6889142f31cb0a5a8b1a208a3e7ed2d133eb89372126fb89b97761f.
[taskcluster 2019-04-12 03:49:46.974Z] === Task Starting ===
[setup 2019-04-12T03:49:47.440Z] run-task started in /builds/worker
[cache 2019-04-12T03:49:47.442Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[cache 2019-04-12T03:49:47.442Z] cache /builds/worker/tooltool-cache exists; requirements: gid=1000 uid=1000 version=1
[volume 2019-04-12T03:49:47.442Z] changing ownership of volume /builds/worker/.cache to 1000:1000
[volume 2019-04-12T03:49:47.442Z] volume /builds/worker/checkouts is a cache
[volume 2019-04-12T03:49:47.442Z] volume /builds/worker/tooltool-cache is a cache
[volume 2019-04-12T03:49:47.442Z] changing ownership of volume /builds/worker/workspace to 1000:1000
[setup 2019-04-12T03:49:47.442Z] running as worker:worker
[fetches 2019-04-12T03:49:47.442Z] fetching artifacts
[fetches 2019-04-12T03:49:47.442Z] executing ['/usr/bin/python3', '-u', '/builds/worker/bin/fetch-content', 'task-artifacts']
Traceback (most recent call last):
  File "/builds/worker/bin/fetch-content", line 443, in <module>
    sys.exit(main())
  File "/builds/worker/bin/fetch-content", line 439, in main
    return args.func(args)
  File "/builds/worker/bin/fetch-content", line 393, in command_task_artifacts
    root_url = os.environ['TASKCLUSTER_ROOT_URL']
  File "/usr/lib/python3.5/os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'TASKCLUSTER_ROOT_URL'

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I'm guessing that they're running a fairly old version of whatever worker implementation they use. As a workaround, you can set TASKCLUSTER_ROOT_URL in task.payload.env, but the better fix will be an upgrade of the worker.

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1543893 - Set TASKCLUSTER_ROOT_URL on workers that don't set it.

Comment 3

[Tom Prince [:tomprince]]

Any chance we could get these workers updated to use a new enough (or patched) worker that sets this?

Comment 4

[Chris Cooper [:coop] (he/him)]

Looking at https://github.com/taskcluster/taskcluster-infrastructure/blob/master/modules/docker-worker/main.tf#L131, I see TASKCLUSTER_ROOT_URL being set, but I confess I don't know whether that's actually propagated into the worker.

Wander: what do we need to do to make this right?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

The required change landed in https://github.com/taskcluster/docker-worker/pull/418, if that helps.

Comment 6

[Wander Lairson Costa]

(In reply to Chris Cooper [:coop] pronoun: he from comment #4)

Looking at https://github.com/taskcluster/taskcluster-infrastructure/blob/master/modules/docker-worker/main.tf#L131, I see TASKCLUSTER_ROOT_URL being set, but I confess I don't know whether that's actually propagated into the worker.

Wander: what do we need to do to make this right?

it is fixed in custom images. Once it lands, we are good.

Comment 7

[Tom Prince [:tomprince]]

:wcosta Do you have an ETA for this landing? It is blocking other worker.

Comment 8

[Wander Lairson Costa]

(In reply to Tom Prince [:tomprince] from comment #7)

:wcosta Do you have an ETA for this landing? It is blocking other worker.

I am close to getting new images done, but I am busy right now with GCP migration and ops hand off.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:glandium, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 10

[Mike Hommey [:glandium]]

Should I just land the workaround patch, then?

Comment 11

[Wander Lairson Costa]

(In reply to Mike Hommey [:glandium] from comment #10)

Should I just land the workaround patch, then?

Yes, please. We are currently evaluating AWS bare metal as a replacement for packet.net.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/30f7688e2759
Set TASKCLUSTER_ROOT_URL on workers that don't set it. r=tomprince

Comment 13

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/30f7688e2759