Closed Bug 1383121 Opened 7 years ago Closed 7 years ago

mark the release as shipped builder fails in 55.0b{11,12} with 401

Categories

(Release Engineering :: Release Automation: Other, defect, P1)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mtabara, Assigned: mtabara)

Details

(Whiteboard: [releaseduty])

Attachments

(2 files)

Most likely fallback from secrets rotation:

08:59:06     INFO - [mozharness: 2017-07-21 15:59:06.724079Z] Finished mark-as-shipped step (failed)
08:59:06    FATAL - Uncaught exception: Traceback (most recent call last):
08:59:06    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/scripts/mozharness/base/script.py", line 2068, in run
08:59:06    FATAL -     self.run_action(action)
08:59:06    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/scripts/mozharness/base/script.py", line 2007, in run_action
08:59:06    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
08:59:06    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/scripts/mozharness/base/script.py", line 1947, in _possibly_run_method
08:59:06    FATAL -     return getattr(self, method_name)()
08:59:06    FATAL -   File "scripts/scripts/release/postrelease_mark_as_shipped.py", line 106, in mark_as_shipped
08:59:06    FATAL -     status='shipped', shippedAt=shipped_at)
08:59:06    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/shipitapi/shipitapi.py", line 110, in update
08:59:06    FATAL -     url_template_vars=url_template_vars).content
08:59:06    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/shipitapi/shipitapi.py", line 61, in request
08:59:06    FATAL -     res.raise_for_status()
08:59:06    FATAL -   File "/builds/slave/rel-m-beta-fx_mark_as_shipped-/build/venv/lib/python2.7/site-packages/requests/models.py", line 909, in raise_for_status
08:59:06    FATAL -     raise HTTPError(http_error_msg, response=self)
08:59:06    FATAL - HTTPError: 401 Client Error: Authorization Required for url: https://ship-it.mozilla.org/csrf_token
08:59:06    FATAL - Running post_fatal callback...
08:59:06    FATAL - Exiting -1
08:59:06     INFO - Running post-run listener: copy_logs_to_upload_dir
08:59:06     INFO - Copying logs to upload dir...
08:59:06     INFO - mkdir: /builds/slave/rel-m-beta-fx_mark_as_shipped-/build/upload/logs


Firefox - https://tools.taskcluster.net/groups/PKW6AX1_Ty6Aw4tcB_CV6A/tasks/R2A-ZhE1QhurJvab3xp_Gw/runs/4/logs/public%2Flogs%2Flive_backing.log.gz
DevEdition - https://tools.taskcluster.net/groups/9wkcaGyMTCC9UomQmsuyCQ/tasks/a0gLwmguQm2r6nC_-l2_bA/runs/5/logs/public%2Flogs%2Flive_backing.log.gz
I'll perform the Mark the release as shipped manually so that I don't block the related changes to the 55.0b11 down the stream. But I'll keep on debugging here. I can always rerun one of the jobs and result in a no-op if 401 is fixed.
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #1)
> I'll perform the Mark the release as shipped manually so that I don't block
> the related changes to the 55.0b11 down the stream. But I'll keep on
> debugging here. I can always rerun one of the jobs and result in a no-op if
> 401 is fixed.

Did the same for beta/devedition/fennec 55.012.

Major caveat: because of Fennec's bug 1347635, we can't actually mark this release as shipped in Ship-it.
So there' couple of workarounds:

1. Solve this bug and fix the key rotations
2. Ask :rail to mark it from ship-it Database
3. Solve bug 1347635 and also do 2).

I'll go ahead with 1) and chase those missing credentials and rerun the builder job.
Assignee: nobody → mtabara
Priority: -- → P1
Whiteboard: [releaseduty]
Accounts for releaserunner - shipit communication were changed in bug 1368164. I checked both bm83 and bm85 and the credentials look correct. Taking a look at the full log [1], it seems the user it tries to connect to 'https://ship-it.mozilla.org' is still 'ffxbld'. That likely because I forgot to change [2] when rotating the keys. Sorry for that. 

[1] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-noarch/release-mozilla-beta-firefox_mark_as_shipped-bm91-build1-build12.txt.gz
[2] https://dxr.mozilla.org/build-central/source/puppet/modules/buildmaster/templates/BuildSlaves-build.py.erb#40-41
Attachment #8890243 - Flags: review?(mtabara)
Attachment #8890243 - Flags: review?(mtabara) → review+
The reasoning behind that is this:
1. The main script that performs the Ship-it API call is this[1]
2. The script gets configs from here[2] (depending on the type of release and if its production/staging) - all config files that contain the "postrelease" string within
3. The BB master builds up the BuildSlaves-build.py file based on this puppet template[3] and populates it. It then passes this down the stream to the slaves renaming it as "oauth.txt". See [4] for this operation
4. `mark release as shipped` script gets this file as config here[5]
5. But because the ship-it-credentials are a dict encompassing both prod/staging[6] creds, it wouldn't know which one to pick. Reminder this script is being used both in staging and production.
6. Because of this reason, we needed to add a flipping variable in the config files from [2]. So for example the Firefox beta configs for this file, which lie here[7] tell us which user is to be used. If we are production, we'll use this[8], otherwise [9] for staging.
7. The logic in the script will then, depending on this config, flip[10] between them 

So long story short:
* we need to land the patch from this[11] comment to make the change available on all Build Slaves with the proper credentials
* but we also need to update the username in all the configs[12][13][14][15][16]

Obs:
a) Once we land the puppet patch, we need to wait for new golden AMI to be generated and new spot instances to come up with refreshed creds
b) tree changes neeed to be uplifted in the tree to beta. So this means, there's no way we can fix this and rerun the failing builder from 55.0b12. We'll need to do that manually in the database.

[1]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/postrelease_mark_as_shipped.py
[2]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases
[3]: https://hg.mozilla.org/build/puppet/file/tip/modules/buildmaster/templates/BuildSlaves-build.py.erb
[4]: https://hg.mozilla.org/build/buildbotcustom/file/tip/process/factory.py#l4392
[5]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/postrelease_mark_as_shipped.py#l44
[6]: https://hg.mozilla.org/build/puppet/file/tip/modules/buildmaster/templates/BuildSlaves-build.py.erb#l35
[7]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/postrelease_firefox_beta.py
[8]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/postrelease_firefox_beta.py#l17
[9]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/dev_postrelease_firefox_beta.py#l19
[10]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/postrelease_mark_as_shipped.py#l95
[11]: https://bugzilla.mozilla.org/show_bug.cgi?id=1383121#c4
[12]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/postrelease_firefox_release.py#l21
[13]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/postrelease_firefox_esr52.py
[14]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/postrelease_firefox_beta.py
[15]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/dev_postrelease_firefox_beta.py#l19
[16]: https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/configs/releases/dev_postrelease_firefox_beta.py#l19
[17]: https://tools.taskcluster.net/groups/3ocvd7YlR0awLgRzyuohRA/tasks/j61KB3qLRs225BklXcRXCQ/details
Summary: mark the release as shipped builder in 55.0b11 fails with 401 → mark the release as shipped builder fails in 55.0b{11,12} with 401
@rail: can you please do a cleanup in Ship-it DB so that:
1. Fennec-55.0b12-build1 first marked as "shipped" and then "Completed". Or the other way around, but it needs to be marked as shipped to produce the desired results in http://product-details.mozilla.org/
2. the rest of them just completed so that they're moved from that "reviewed" section

Thanks!
Flags: needinfo?(rail)
Comment on attachment 8890301 [details]
Bug 1383121 - fix username in shipit builder in automation configs.  a=release DONTBUILD

https://reviewboard.mozilla.org/r/161428/#review166726

Looks good!
Attachment #8890301 - Flags: review?(aselagea) → review+
ran `update fennec_release set complete=1, status='shipped' where name='Fennec-55.0b12-build1';`

mysql> select status, complete, ready from fennec_release where name='Fennec-55.0b12-build1';
+---------+----------+-------+
| status  | complete | ready |
+---------+----------+-------+
| shipped |        1 |     1 |
+---------+----------+-------+
1 row in set (0.00 sec)
Flags: needinfo?(rail)
(In reply to Rail Aliiev [:rail] ⌚️ET - PTO in August from comment #9)
> ran `update fennec_release set complete=1, status='shipped' where
> name='Fennec-55.0b12-build1';`
> 
> mysql> select status, complete, ready from fennec_release where
> name='Fennec-55.0b12-build1';
> +---------+----------+-------+
> | status  | complete | ready |
> +---------+----------+-------+
> | shipped |        1 |     1 |
> +---------+----------+-------+
> 1 row in set (0.00 sec)

Awesome, thanks!
Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/36f12c9341cd
fix username in shipit builder in automation configs. r=aselagea a=release DONTBUILD
Comment on attachment 8890301 [details]
Bug 1383121 - fix username in shipit builder in automation configs.  a=release DONTBUILD

https://hg.mozilla.org/integration/mozilla-inbound/rev/36f12c9341cd2ac0f8a7efcd00d124f67df4f3a0
https://hg.mozilla.org/releases/mozilla-beta/rev
/c06ae6f08dbeee5dfd10ddae3e26ce9fbfd7a2ca
Attachment #8890301 - Flags: checked-in+
Keywords: leave-open
Everything is landed. We'll wait for 55.0b13 on Friday to ship and confirm that we've changed everything accordingly.
All good. Firefox 55.0b13 mark-release-as-shipped builders worked like a charm!
On a separate note, thanks :nthomas for shipit db cleanup earlier today!
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.