Closed Bug 1446479 Opened 6 years ago Closed 6 years ago

Windows tasks shouldn't try to use the relengapi-proxy

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(firefox59 fixed, firefox60 fixed, firefox61 fixed)

RESOLVED FIXED
mozilla61
Tracking Status
firefox59 --- fixed
firefox60 --- fixed
firefox61 --- fixed

People

(Reporter: jlorenzo, Assigned: Callek)

Details

Attachments

(1 file)

For a few days, gecko-3-b-win2012 workers haven't managed to access releng API. This is a permanent failure on this branch. Failure shows as [1]:

> 13:39:39     INFO -   8:22.38 HTTPConnectionPool(host='relengapi', port=80): Max retries exceeded with url: /tooltool/sha512/18700889e6b5e81613b9cf57ce4e0d46a6ee45bb4c5c33bae2604a5275326128775b8a032a1eb178c5db973746d565340c4e36d98375789e1d5bd836ab16ba58 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x0000000004291160>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))
> 13:39:39     INFO -   8:22.38 Failed to download vs2017_15.4.2.zip
> 13:39:39    ERROR - Return code: 1

This looks very similar to the DNS problem that caused {firefox,devedition} 60.0b4 to fail, yesterday[2]

Here are the timestamps of the first noticed changes:
* First broken: https://treeherder.mozilla.org/#/jobs?repo=maple&revision=6b5d541cd9696a51afb3a5bc5e583d307c475339&filter-searchStr=N1&selectedJob=167666384
* Last working: https://treeherder.mozilla.org/#/jobs?repo=maple&revision=1e0275c17b6cf25563de0f6af4f60c57edc0400a&filter-searchStr=N1&selectedJob=167446802

I didn't see any in-tree changes on maple that would cause such issue.


[1] https://treeherder.mozilla.org/logviewer.html#?job_id=167666384&repo=maple&lineNumber=679
[2] For instance: https://tools.taskcluster.net/groups/MIUq4oRVRXGMfyduHB3W3Q/tasks/HgM-lj2eTKi_7zNcmOYu2w/runs/0/logs/public%2Flogs%2Flive_backing.log#L806
Turns out is actually a task configuration bug... taking
Assignee: relops → bugspam.Callek
Component: RelOps → Task Configuration
Product: Infrastructure & Operations → Firefox Build System
QA Contact: klibby
Version: unspecified → Trunk
Summary: gecko-3-b-win2012 on branch maple can't access relengapi → Windows tasks shouldn't try to use the relengapi-proxy
So, it seems we've (since I landed windows l10n in taskcluster) been using http://relengapi/tooltool/ as the tooltool url, this has only been working because if we have the relevent file in local cache we don't query against tooltool.

With TC's windows spinup process it fetches a set of tooltool files to cache locally, and this has been saving us, meaning we're very lucky.

However maple (beta-based) and central have just diverged in manifests, because central now uses a newer VS version than what we have on maple.

This patch will fix that to use the new tooltool location that we currently use for Windows builds, and aligns the repackage jobs to use the newer url too.
Comment on attachment 8959684 [details]
Bug 1446479 - Windows tasks shouldn't try to use the relengapi-proxy. r=tomprince

Tom Prince [:tomprince] has approved the revision.

https://phabricator.services.mozilla.com/D743
Attachment #8959684 - Flags: review+
Comment on attachment 8959684 [details]
Bug 1446479 - Windows tasks shouldn't try to use the relengapi-proxy. r=tomprince

Approval Request Comment
[Feature/Bug causing the regression]: Switch to taskcluster for l10n
[User impact if declined]: Potential broken beta's/release builds
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: Landed to inbound just now
[Needs manual test from QE? If yes, steps to reproduce]: No
[List of other uplifts needed for the feature/fix]: No other uplifts other than those requested here
[Is the change risky?]: No
[Why is the change risky/not risky?]: Not risky because this is what windows builds are using now.
[String changes made/needed]: None

This was only noticed now because the cache on the workers was preventing us from noticing the problem.
Attachment #8959684 - Flags: approval-mozilla-release?
Attachment #8959684 - Flags: approval-mozilla-beta?
Pushed by Callek@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/30458407e1e4
Windows tasks shouldn't try to use the relengapi-proxy. r=tomprince
Comment on attachment 8959684 [details]
Bug 1446479 - Windows tasks shouldn't try to use the relengapi-proxy. r=tomprince

This could probably just land a=release, but approving it officially anyway. This bug made yesterday's Beta L10n jobs for the chemspill a giant pain, so we definitely want this before we have to ship any other releases.
Attachment #8959684 - Flags: approval-mozilla-release?
Attachment #8959684 - Flags: approval-mozilla-release+
Attachment #8959684 - Flags: approval-mozilla-beta?
Attachment #8959684 - Flags: approval-mozilla-beta+
Requesting checkin-needed for this patch to land on beta and release, should be a simple graft.
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/30458407e1e4
Status: NEW → RESOLVED
Closed: 6 years ago
Keywords: checkin-needed
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: