Closed Bug 1414678 Opened 2 years ago Closed 2 years ago

Android L10n builds fails to fetch secrets

Categories

(Firefox Build System :: Android Studio and Gradle Integration, defect, blocker)

defect
Not set
blocker

Tracking

(firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox58 --- fixed

People

(Reporter: aryx, Assigned: nalexander)

References

Details

(Whiteboard: [stockwell infra])

Attachments

(1 file)

Fallout from bug 1411688. The last 4 nightly builds failed, 2 of them for this reason, 2 because of broken localizations.

https://treeherder.mozilla.org/logviewer.html#?job_id=142138522&repo=mozilla-central

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=14fd26761bc4d10c5334abe50d7b6f3a5908f08d&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

[task 2017-11-04T11:06:02.144Z] 11:06:02     INFO - fetching secret project/releng/gecko/build/level-1/gapi.data from API
[task 2017-11-04T11:06:02.562Z] 11:06:02     INFO - [mozharness: 2017-11-04 11:06:02.562084Z] Finished get-secrets step (failed)
[task 2017-11-04T11:06:02.563Z] 11:06:02    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2017-11-04T11:06:02.563Z] 11:06:02    FATAL -   File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 2059, in run
[task 2017-11-04T11:06:02.563Z] 11:06:02    FATAL -     self.run_action(action)
[task 2017-11-04T11:06:02.564Z] 11:06:02    FATAL -   File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 1998, in run_action
[task 2017-11-04T11:06:02.564Z] 11:06:02    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2017-11-04T11:06:02.564Z] 11:06:02    FATAL -   File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/base/script.py", line 1938, in _possibly_run_method
[task 2017-11-04T11:06:02.564Z] 11:06:02    FATAL -     return getattr(self, method_name)()
[task 2017-11-04T11:06:02.565Z] 11:06:02    FATAL -   File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/mozilla/secrets.py", line 68, in get_secrets
[task 2017-11-04T11:06:02.565Z] 11:06:02    FATAL -     secret = self._fetch_secret(secret_name)
[task 2017-11-04T11:06:02.565Z] 11:06:02    FATAL -   File "/builds/worker/workspace/build/src/testing/mozharness/mozharness/mozilla/secrets.py", line 25, in _fetch_secret
[task 2017-11-04T11:06:02.565Z] 11:06:02    FATAL -     res = urllib2.urlopen(url)
[task 2017-11-04T11:06:02.566Z] 11:06:02    FATAL -   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
[task 2017-11-04T11:06:02.566Z] 11:06:02    FATAL -     return opener.open(url, data, timeout)
[task 2017-11-04T11:06:02.566Z] 11:06:02    FATAL -   File "/usr/lib/python2.7/urllib2.py", line 435, in open
[task 2017-11-04T11:06:02.566Z] 11:06:02    FATAL -     response = meth(req, response)
[task 2017-11-04T11:06:02.567Z] 11:06:02    FATAL -   File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
[task 2017-11-04T11:06:02.567Z] 11:06:02    FATAL -     'http', request, response, code, msg, hdrs)
[task 2017-11-04T11:06:02.567Z] 11:06:02    FATAL -   File "/usr/lib/python2.7/urllib2.py", line 473, in error
[task 2017-11-04T11:06:02.568Z] 11:06:02    FATAL -     return self._call_chain(*args)
[task 2017-11-04T11:06:02.568Z] 11:06:02    FATAL -   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
[task 2017-11-04T11:06:02.568Z] 11:06:02    FATAL -     result = func(*args)
[task 2017-11-04T11:06:02.568Z] 11:06:02    FATAL -   File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
[task 2017-11-04T11:06:02.569Z] 11:06:02    FATAL -     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
[task 2017-11-04T11:06:02.569Z] 11:06:02    FATAL - HTTPError: HTTP Error 403: Forbidden
[task 2017-11-04T11:06:02.569Z] 11:06:02    FATAL - Running post_fatal callback...
[task 2017-11-04T11:06:02.569Z] 11:06:02    FATAL - Exiting -1
Flags: needinfo?(nalexander)
This must be some SCM level interaction not seen on try or on oak.  How is that even possible?  oak is the same as central for levels, right?  I can try to understand what scopes the job ran with after a meeting. 

Callek: this isn't really single-locale specific; it's mozharness and configuration goop required to make the single-locale build step have the same secrets as the underlying Nightly build step.  Any thoughts?
Flags: needinfo?(nalexander) → needinfo?(bugspam.Callek)
(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #0)
> [task 2017-11-04T11:06:02.144Z] 11:06:02     INFO - fetching secret
> project/releng/gecko/build/level-1/gapi.data from API

Problem 1:  This shouldn't be fetching `level-1` secret in mozilla-central.

Problem 2: Not able to fetch that level-1 secret (scopes?)


This (doing level 1) is due to l10n jobs not passing `--level` to the mozharness script to satisfy https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/secrets.py#51 because its not a recognized config var.  This will need to be fixed.


(In reply to Nick Alexander :nalexander from comment #1)
> This must be some SCM level interaction not seen on try or on oak.  How is
> that even possible?  oak is the same as central for levels, right?  I can
> try to understand what scopes the job ran with after a meeting. 
> 
> Callek: this isn't really single-locale specific; it's mozharness and
> configuration goop required to make the single-locale build step have the
> same secrets as the underlying Nightly build step.  Any thoughts?

Can you point me at a try and a oak run of this code, so I can see whats different and how it worked there?
Flags: needinfo?(bugspam.Callek) → needinfo?(nalexander)
(In reply to Justin Wood (:Callek) from comment #2)
> (In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on
> intermittent or backout) from comment #0)
> > [task 2017-11-04T11:06:02.144Z] 11:06:02     INFO - fetching secret
> > project/releng/gecko/build/level-1/gapi.data from API
> 
> Problem 1:  This shouldn't be fetching `level-1` secret in mozilla-central.
> 
> Problem 2: Not able to fetch that level-1 secret (scopes?)
> 
> 
> This (doing level 1) is due to l10n jobs not passing `--level` to the
> mozharness script to satisfy
> https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/
> mozilla/secrets.py#51 because its not a recognized config var.  This will
> need to be fixed.
> 
> 
> (In reply to Nick Alexander :nalexander from comment #1)
> > This must be some SCM level interaction not seen on try or on oak.  How is
> > that even possible?  oak is the same as central for levels, right?  I can
> > try to understand what scopes the job ran with after a meeting. 
> > 
> > Callek: this isn't really single-locale specific; it's mozharness and
> > configuration goop required to make the single-locale build step have the
> > same secrets as the underlying Nightly build step.  Any thoughts?
> 
> Can you point me at a try and a oak run of this code, so I can see whats
> different and how it worked there?

Sure!  I had a happy `fr` build at:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=df5a389065620213a72a10a12634ec479d34d373

Now I can't find green oak Nx jobs, but I did 90% of the development on oak, so I'm confused.  Maybe I made a long-running error?
Flags: needinfo?(nalexander)
On the manually retriggered job at

https://tools.taskcluster.net/groups/YLJqsV99RZG4LbPRg1Eb4Q/tasks/YLJqsV99RZG4LbPRg1Eb4Q/runs/0

I see

[task 2017-11-06T22:24:29.270Z] 22:24:29     INFO - [mozharness: 2017-11-06 22:24:29.270821Z] Running get-secrets step.
[task 2017-11-06T22:24:29.271Z] 22:24:29     INFO - Running main action method: get_secrets
[task 2017-11-06T22:24:29.271Z] 22:24:29     INFO - fetching secret project/releng/gecko/build/level-3/gapi.data from API
[task 2017-11-06T22:24:29.444Z] 22:24:29     INFO - fetching secret project/releng/gecko/build/level-3/mozilla-fennec-geoloc-api.key from API
[task 2017-11-06T22:24:29.548Z] 22:24:29     INFO - fetching secret project/releng/gecko/build/level-3/adjust-sdk.token from API
[task 2017-11-06T22:24:29.665Z] 22:24:29     INFO - fetching secret project/releng/gecko/build/level-3/adjust-sdk-beta.token from API
[task 2017-11-06T22:24:29.835Z] 22:24:29     INFO - fetching secret project/releng/gecko/build/level-3/leanplum-sdk-release.token from API
[task 2017-11-06T22:24:29.948Z] 22:24:29     INFO - fetching secret project/releng/gecko/build/level-3/leanplum-sdk-beta.token from API
[task 2017-11-06T22:24:30.056Z] 22:24:30     INFO - fetching secret project/releng/gecko/build/level-3/leanplum-sdk-nightly.token from API
[task 2017-11-06T22:24:30.161Z] 22:24:30     INFO - fetching secret project/releng/gecko/build/level-3/pocket-api-release.token from API
[task 2017-11-06T22:24:30.264Z] 22:24:30     INFO - fetching secret project/releng/gecko/build/level-3/pocket-api-beta.token from API
[task 2017-11-06T22:24:30.381Z] 22:24:30     INFO - fetching secret project/releng/gecko/build/level-3/pocket-api-nightly.token from API
[task 2017-11-06T22:24:30.493Z] 22:24:30     INFO - [mozharness: 2017-11-06 22:24:30.493107Z] Finished get-secrets step (success)
[task 2017-11-06T22:24:30.493Z] 22:24:30     INFO - [mozharness: 2017-11-06 22:24:30.493445Z] Skipping clobber step.
[task 2017-11-06T22:24:30.493Z] 22:24:30     INFO - [mozharness: 2017-11-06 22:24:30.493611Z] Skipping pull step.
[task 2017-11-06T22:24:30.493Z] 22:24:30     INFO - [mozharness: 2017-11-06 22:24:30.493774Z] Running clone-locales step.

Will push patch for Callek to review.
Whiteboard: [stockwell needswork]
Whiteboard: [stockwell needswork] → [stockwell infra]
Comment on attachment 8925702 [details]
Bug 1414678 - Pass through MOZ_SCM_LEVEL, so that secret fetching works.

https://reviewboard.mozilla.org/r/196828/#review202392

Said in IRC, that this looks pretty good, but we'll need to add the option support to the desktop script too.
Attachment #8925702 - Flags: review?(bugspam.Callek)
Try is looking OK so far at https://treeherder.mozilla.org/#/jobs?repo=try&revision=16d5a5223c66a35b1b781fe387a3ca2b23b4dea6.

flake8 failure addressed in review request.
Assignee: nobody → nalexander
Status: NEW → ASSIGNED
Comment on attachment 8925702 [details]
Bug 1414678 - Pass through MOZ_SCM_LEVEL, so that secret fetching works.

https://reviewboard.mozilla.org/r/196828/#review202488
Attachment #8925702 - Flags: review?(bugspam.Callek) → review+
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3f07fe29ae82
Pass through MOZ_SCM_LEVEL, so that secret fetching works. r=Callek
https://hg.mozilla.org/mozilla-central/rev/3f07fe29ae82
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 58
Product: Firefox for Android → Firefox Build System
Target Milestone: Firefox 58 → mozilla58
You need to log in before you can comment on or make changes to this bug.