1760699 - Permanent fenix browsertime [taskcluster:error] [mounts] Could not fetch from url http://taskcluster/api/queue/v1/task/ ... Get http://taskcluster/api/queue/v1/task/ ... dial tcp: lookup taskcluster on 127.0.0.11:53: no such host

Status: RESOLVED FIXED

(Release Engineering :: Firefox-CI Administration, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

All browsertime tasks on fenix fail. Regression from https://mozilla-hub.atlassian.net/browse/RELENG-805 (?)

Push: https://treeherder.mozilla.org/jobs?repo=fenix&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cusercancel&revision=1647d935912414fa9de5cc2d8cb36aabd0468572&selectedTaskRun=QaOCTTUoSXeew4ygzE062w.0
Failure log: https://treeherder.mozilla.org/logviewer?job_id=371803776&repo=fenix

[taskcluster 2022-03-21T19:59:41.583Z] [mounts] Creating directory /builds/task_164789277847811 with permissions 0700
[taskcluster 2022-03-21T19:59:41.583Z] [mounts] Copying downloads/EJ4TrHq6QIWJOYvk_n1a2w to /builds/task_164789277847811/test-linux.sh
[taskcluster 2022-03-21T19:59:41.632Z] [mounts] Downloading url http://taskcluster/api/queue/v1/task/IiBV4T77Qb2bpBI_KizSiw/artifacts/public/run-task to downloads/LK649XcTRE6fJMbzNykHHw
[taskcluster:warn 2022-03-21T19:59:41.645Z] [mounts] Download of url http://taskcluster/api/queue/v1/task/IiBV4T77Qb2bpBI_KizSiw/artifacts/public/run-task failed on this attempt: Get http://taskcluster/api/queue/v1/task/IiBV4T77Qb2bpBI_KizSiw/artifacts/public/run-task: dial tcp: lookup taskcluster on 127.0.0.11:53: no such host

Comment 1

[Aki Sasaki (not active)]

We probably have to enable the taskclusterProxy. (task.payload.features["taskclusterProxy"] -> True)

Comment 2

[Julien Cristau [:jcristau]]

Attachment: Fenix Pull Request

Comment 3

[Heitor Neiva [:hneiva]]

Attachment: Bug 1760699 - Don't use proxy for run-task on generic-workers r?#releng-reviewers!

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/fe3d3d607137
Don't use proxy for run-task on generic-workers r=releng-reviewers,ahal

Comment 5

[Julien Cristau [:jcristau]]

AIUI the first attempt didn't actually fix this.

Comment 6

[Heitor Neiva [:hneiva]]

(In reply to Julien Cristau [:jcristau] from comment #5)

AIUI the first attempt didn't actually fix this.

The first attempt was just to try and get them moving, but we were unable to get another nightly build going.
So, we released a fixed version of taskgraph and updated fenix to use it.

https://github.com/mozilla-mobile/fenix/pull/24392
Once nightly triggered, this specific error was gone, and browsertime tests were running:
https://treeherder.mozilla.org/jobs?repo=fenix&revision=c64600525dde5314192d9cb6bbaecbe9dff7f458

Comment 7

[Intermittent Failures Robot]

180 failures in 3341 pushes (0.054 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• fenix: 180

Platform and build breakdown:

• android-hw-g5-7-0-arm7-api-16-shippable-qr: 90
  • opt: 90
• android-hw-p2-8-0-android-aarch64-shippable-qr: 90
  • opt: 90

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760699&startday=2022-03-21&endday=2022-03-27&tree=all

Comment 8

[Pulsebot]

Pushed by ffxbld-merge:
https://hg.mozilla.org/releases/mozilla-beta/rev/1664a93d7e49
[fenix] Bug 1760699 - Fix proxy on run-task jobs
https://hg.mozilla.org/releases/mozilla-beta/rev/2a838631f97d
[fenix] Bug 1760699 - Revert proxy settings on browsertime task