Closed Bug 1365219 Opened 7 years ago Closed 7 years ago

Windows Build fail with 403 pulling vs2015u3 zip


(Infrastructure & Operations Graveyard :: CIDuty, task)

Not set


(Not tracked)



(Reporter: cbook, Unassigned)



(Whiteboard: [stockwell infra])

We have massive Windows build Problems currently and only several retrigger helps.

Its always like

03:48:12     INFO -   0:07.39 attempt 1/5
03:48:12     INFO -   0:07.39 Downloading to temporary location c:\builds\tooltool_cache\babc414ffc0457d27f5a1ed24a8e4873afbe2f1c1a4075469a27c005e1babc3b2a788f643f825efedff95b79686664c67ec4340ed535487168a3482e68559bc7
03:48:12     INFO -   0:07.45 403 Client Error: FORBIDDEN for url:
03:48:12     INFO -   0:07.45 Failed to download

might be related to
more and more failures in windows builds, so closing inbound and autoland
Severity: critical → blocker
I've reverted b9c1aed49990b4a7e7ad28a90e030219d2634f5f to see if that will help.
Depends on: 1362356
I believe is the only INTERNAL download in that releng.manifest -- the rest are visibility=PUBLIC.

(The manifest entry for mozmake.exe does not specify a visibility, but reports that mozmake.exe is PUBLIC.)

An attempt to download an INTERNAL file without an appropriate token will result in a 403 -- consistent with the ideas in and around
The latest theory on this is that runner is starting before the one of the instantiation scripts is finished copying over data to c:\builds. The job fails because the tokens aren't there yet, then runner reboots the machine, so the copy of the keys never finishes. That instance is bad from the start and keeps burning jobs. Instances that do not pick up jobs before the copy is finished have the keys and operate normally.

Markco is going to try creating a sempahore file so that runner doesn't start until the token copy has finished
Because we're still having issues with AMI generation, we've hand modified the AMI from May 2nd and taken a snapshot for b-2008 and y-2008. These have been copied over to usw2 now as well, so any new instantiations should happen from the AMIs modified to look for he semaphore.
We think this issue (and a host of others) may be solved now. There's a lengthy explanation in the blocking bug.
Whiteboard: [stockwell infra]
tjr's issue was unrelated.
Closed: 7 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.