1365219 - Windows Build fail with 403 pulling vs2015u3 zip

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

We have massive Windows build Problems currently and only several retrigger helps.

Its always like

ownloading vs2015u3.zip
03:48:12     INFO -   0:07.39 attempt 1/5
03:48:12     INFO -   0:07.39 Downloading to temporary location c:\builds\tooltool_cache\babc414ffc0457d27f5a1ed24a8e4873afbe2f1c1a4075469a27c005e1babc3b2a788f643f825efedff95b79686664c67ec4340ed535487168a3482e68559bc7
03:48:12     INFO -   0:07.45 403 Client Error: FORBIDDEN for url: https://api.pub.build.mozilla.org/tooltool/sha512/babc414ffc0457d27f5a1ed24a8e4873afbe2f1c1a4075469a27c005e1babc3b2a788f643f825efedff95b79686664c67ec4340ed535487168a3482e68559bc7
03:48:12     INFO -   0:07.45 Failed to download vs2015u3.zip

https://treeherder.mozilla.org/logviewer.html#?job_id=99405429&repo=mozilla-inbound&lineNumber=1775

might be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1362356#c25

Comment 1

[Carsten Book [:Tomcat]]

more and more failures in windows builds, so closing inbound and autoland

Comment 2

[Amy Rich [:arr] [:arich]]

I've reverted b9c1aed49990b4a7e7ad28a90e030219d2634f5f to see if that will help.

Comment 3

[Geoff Brown [:gbrown]]

I believe vs2015u3.zip is the only INTERNAL download in that releng.manifest -- the rest are visibility=PUBLIC.

(The manifest entry for mozmake.exe does not specify a visibility, but api.pub.build.mozilla.org/tooltool reports that mozmake.exe is PUBLIC.)

An attempt to download an INTERNAL file without an appropriate token will result in a 403 -- consistent with the ideas in and around https://bugzilla.mozilla.org/show_bug.cgi?id=1362356#c25.

Comment 4

[Amy Rich [:arr] [:arich]]

The latest theory on this is that runner is starting before the one of the instantiation scripts is finished copying over data to c:\builds. The job fails because the tokens aren't there yet, then runner reboots the machine, so the copy of the keys never finishes. That instance is bad from the start and keeps burning jobs. Instances that do not pick up jobs before the copy is finished have the keys and operate normally.

Markco is going to try creating a sempahore file so that runner doesn't start until the token copy has finished

Comment 5

[Amy Rich [:arr] [:arich]]

Because we're still having issues with AMI generation, we've hand modified the AMI from May 2nd and taken a snapshot for b-2008 and y-2008. These have been copied over to usw2 now as well, so any new instantiations should happen from the AMIs modified to look for he semaphore.

Comment 6

[Intermittent Failures Robot]

300 failures in 77 pushes (3.896 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 122
* mozilla-central: 81
* mozilla-inbound: 46
* mozilla-beta: 23
* mozilla-release: 13
* mozilla-esr52: 6
* try: 4
* mozilla-esr45: 3
* graphics: 2

Platform breakdown:
* windowsxp: 155
* windows8-64: 137
* windows8-32: 4
* windows2012-64: 2
* windows8-64-devedition: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1365219&startday=2017-05-16&endday=2017-05-16&tree=all

Comment 7

[Amy Rich [:arr] [:arich]]

We think this issue (and a host of others) may be solved now. There's a lengthy explanation in the blocking bug.

Comment 9

[Tom Ritter [:tjr] (OOTO until 4/30 at least)]

Something seems to be a bit screwy still, I got a timeout on a build retriggered after Comment 7 and an entire try run sent after comment 7:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb26c9b90f8a5e3cc1dc5bb46512dc29d900cbc5&selectedJob=99572515
https://treeherder.mozilla.org/#/jobs?repo=try&revision=81a77d7217dda5496904870605905dcfdbb20cba&selectedJob=99625818

Comment 10

[Amy Rich [:arr] [:arich]]

tjr's issue was unrelated.

Comment 11

[Intermittent Failures Robot]

304 failures in 777 pushes (0.391 failures/push) were associated with this bug in the last 7 days. 

This is the #2 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 123
* mozilla-central: 77
* mozilla-inbound: 46
* mozilla-beta: 23
* mozilla-release: 14
* mozilla-esr52: 9
* mozilla-esr45: 6
* try: 4
* graphics: 2

Platform breakdown:
* windowsxp: 156
* windows8-64: 140
* windows8-32: 4
* windows2012-64: 2
* windows8-64-devedition: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1365219&startday=2017-05-15&endday=2017-05-21&tree=all