1385263 - AWFY's downloads of Windows nightly fails

Status: RESOLVED FIXED

(Testing Graveyard :: AWFY, defect)

Description

[Armen [:armenzg]]

Support for nightly builds was added recently in https://github.com/mozilla/arewefastyet/pull/141/files.
We switched Windows nightly builds from Buildbot to TaskCluster this week.
I assume it is related. I will have to investigate.


Command: python download.py --repo mozilla-central -o ~/repos/mozilla-central-nightly/ -c 64bit -b nightly

*******************************************************************************
                                   DOWNLOAD
*******************************************************************************
Downloading for repository mozilla-central
Fetching JSON at https://treeherder.mozilla.org/api/project/mozilla-central/resultset/?count=10
Fetching JSON at https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.nightly.latest.firefox.win64-opt
Fetching JSON at https://public-artifacts.taskcluster.net/HQ4GcWG1T1Gmci_rxGkisQ/0/public/build/buildbot_properties.json
Traceback (most recent call last):
  File "download.py", line 379, in <module>
    options.cset, options.buildtype)
  File "download.py", line 51, in download_for_repo
    urls = creator.find(cset, buildtype=buildtype)
  File "/home/mozilla/arewefastyet/slave/url_creator.py", line 19, in find
    urls = self.latest(**kwargs)[0:5]
  File "/home/mozilla/arewefastyet/slave/url_creator.py", line 136, in latest
    urls = self._url_for_revision(revision, buildtype)
  File "/home/mozilla/arewefastyet/slave/url_creator.py", line 235, in _url_for_revision
    self._platform())]
  File "/home/mozilla/arewefastyet/slave/url_creator.py", line 94, in latest_nightly_url
    return cls._properties(task_id)['packageUrl']
  File "/home/mozilla/arewefastyet/slave/url_creator.py", line 88, in _properties
    return utils.fetch_json(url)['properties']
  File "/home/mozilla/arewefastyet/slave/utils.py", line 212, in fetch_json
    response = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

Comment 1

[Justin Wood (:Callek)]

Ok, without knowing much about AWFY heres my take/info-dump:

The move to taskcluster results in filenames that are "simple" (like OSX and Linux) rather than using the firefox-.*.zip format we had before.

The TC index to find relevant files changed too.... (across a few indexes)

* https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.nightly.latest.firefox.win64-opt
  ** Will be roughly like whats in buildbot, but *everything* is unsigned in there.
* https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.signed-nightly.nightly.latest.firefox.win64-opt
  ** Will have the `target.zip` with signed internals, (and a signed setup and if present setup-stub.exe that your tooling can ignore)
* https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.nightly.latest.firefox/win64-nightly-repackage
  ** Will have the installer, complete.mar and stub installer (if built), the files themselves are unsigned but internals of them are signed.
* https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.nightly.latest.firefox/win64-nightly-repackage-signing
  ** We sign the installer and stub installer and complete mar file at this stage.

I suspect you want a signed windows build for AWFY (so second index in my list) but I'm unsure.

Comment 2

[Armen [:armenzg]]

Attachment: Bug 1385263 - Fix for TaskCluster migration

Comment 3

[Armen [:armenzg]]

bc, would you mind testing this change on the reference platform?
The tests pass for this change, however, I’m having trouble unzipping on my Win64 VM (I sent my reference laptop back)

git clone https://github.com/armenzg/arewefastyet.git
cd arewefastyet/slave
git checkout fix_nightly
python download.py --repo mozilla-central -o ~/repos/mozilla-central-nightly/ -c 64bit -b nightly

Comment 4

[Justin Wood (:Callek)]

Comment on attachment 8891370 [details] [review]
Bug 1385263 - Fix for TaskCluster migration

r+ with nits on the PR https://github.com/mozilla/arewefastyet/pull/151#pullrequestreview-52955045

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: log 1

>> Executing in /home/bclary/repos/mozilla-central-nightly: unzip target.zip

attempts to download

https://public-artifacts.taskcluster.net/DLP8cA6aTJO6I11f7_zJ7Q/0/public/build/target.json

but there is no such file in the artifacts for that build:

https://tools.taskcluster.net/groups/OKoqJx-YTTO4TvmeEM_z6g/tasks/DLP8cA6aTJO6I11f7_zJ7Q/runs/0/artifacts 

Looking at the pull request, I'm not sure I can merge from the github ui. maybe from the command line though. But I don't think we are ready yet.

Comment 6

[Bob Clary [:bc] (inactive)]

Callek: Any ideas on the target.json issue?

Comment 7

[Justin Wood (:Callek)]

https://queue.taskcluster.net/v1/task/DLP8cA6aTJO6I11f7_zJ7Q/runs/0/artifacts/public/build/target.zip is what the UX for task-inspector gets me, since the target.zip indeed does exist. I'm not sure where that log shows a target.json though.

I should note the old index should have the main builds stuff, e.g. if you do need the target.json its on https://tools.taskcluster.net/groups/OKoqJx-YTTO4TvmeEM_z6g/tasks/b-6xjYVXTK6Ges0r6aAiAA/runs/0/artifacts

Comment 8

[Bob Clary [:bc] (inactive)]

r+ Looks good on an win7 vm.

Comment 9

[Armen [:armenzg]]

I've merged the change to the master branch.
IIUC every AWFY run updates the machine to the latest AWFY code, thus, in few hours we should see task 16 completed and get some results.
I've removed task 15 from the queue and forced the scheduling of task 16. Task 14 is currently running and I won't mess with it.

I'm on PTO and back on Monday.

Comment 10

[Armen [:armenzg]]

We're back to normal:
https://arewefastyet.com/#machine=36&view=breakdown&suite=speedometer-misc