Closed Bug 1388323 Opened 8 years ago Closed 7 years ago

Cannot mark 52esr as shipped: HTTPError: 401 Client Error: Authorization Required for url: https://ship-it.mozilla.org/csrf_token

Categories

(Release Engineering :: Release Automation, enhancement, P1)

Product:
Release Engineering
Component:
Release Automation
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mtabara, Unassigned)

References

Details

(Whiteboard: [releaseduty])

Both mark release as shipped[1] and publish to Balrog[2] builder jobs failed today for unauthorized access to Shipit/Database. I strongly suspect that this is fallback from key rotations and we missed some uplifts during mergeday. Might be a problem with the merge script too. For example this config change[3] landed on central/beta but not sure it actually got to release/esr52 as it's currently missing from there[4]. I haven't looked but most likely Balrog builder configs happens for the same reason since we killed the LDAP account and went for dedicated accounts. To workaround this for 55.0 both operations were completed manually. [1]: https://tools.taskcluster.net/groups/UFVbfJ1DRGeWhqEC-WJDcw/tasks/yr8mAcRpSNObgtm_qtjdyg/details [2]: https://tools.taskcluster.net/groups/UFVbfJ1DRGeWhqEC-WJDcw/tasks/JRywvYoaT8uTd3yUqZirAQ/details [3]: https://hg.mozilla.org/mozilla-central/diff/36f12c9341cd/testing/mozharness/configs/releases/postrelease_firefox_esr52.py [4]: https://hg.mozilla.org/releases/mozilla-esr52/file/tip/testing/mozharness/configs/releases/postrelease_firefox_esr52.py
Priority: -- → P3
Whiteboard: [releaseduty]
Assignee: nobody → mtabara
Blocks: 1397840
Priority: P3 → P1
Happened in 52.4.1esr too: * ship-it: https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-esr52-noarch/release-mozilla-esr52-firefox_mark_as_shipped-bm74-build1-build7.txt.gz * balrog: https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-esr52-noarch/release-mozilla-esr52-firefox_schedule_publishing_in_balrog-bm73-build1-build3.txt.gz In my opinion, the ship-it issue looks creds-related. However, balrog is more like a bug in the client. I filed a separate bug (bug 1407174). Let's track the work there. In case the logs disappear before we fix this issue, here's the stacktrace: > 01:42:41 INFO - [mozharness: 2017-10-10 08:42:41.658074Z] Running mark-as-shipped step. > 01:42:41 INFO - Running main action method: mark_as_shipped > 01:42:41 INFO - Mark the release as shipped with 2017-10-10 08:42:41 timestamp > /builds/slave/rel-m-esr52-fx_mark_as_shipped/build/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:334: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings > SNIMissingWarning > /builds/slave/rel-m-esr52-fx_mark_as_shipped/build/venv/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:132: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings > InsecurePlatformWarning > 01:42:41 INFO - [mozharness: 2017-10-10 08:42:41.927495Z] Finished mark-as-shipped step (failed) > 01:42:41 FATAL - Uncaught exception: Traceback (most recent call last): > 01:42:41 FATAL - File "/builds/slave/rel-m-esr52-fx_mark_as_shipped/scripts/mozharness/base/script.py", line 2034, in run > 01:42:41 FATAL - self.run_action(action) > 01:42:41 FATAL - File "/builds/slave/rel-m-esr52-fx_mark_as_shipped/scripts/mozharness/base/script.py", line 1973, in run_action > 01:42:41 FATAL - self._possibly_run_method(method_name, error_if_missing=True) > 01:42:41 FATAL - File "/builds/slave/rel-m-esr52-fx_mark_as_shipped/scripts/mozharness/base/script.py", line 1913, in _possibly_run_method > 01:42:41 FATAL - return getattr(self, method_name)() > 01:42:41 FATAL - File "scripts/scripts/release/postrelease_mark_as_shipped.py", line 106, in mark_as_shipped > 01:42:41 FATAL - status='shipped', shippedAt=shipped_at) > 01:42:41 FATAL - File "/builds/slave/rel-m-esr52-fx_mark_as_shipped/build/venv/lib/python2.7/site-packages/shipitapi/shipitapi.py", line 110, in update > 01:42:41 FATAL - url_template_vars=url_template_vars).content > 01:42:41 FATAL - File "/builds/slave/rel-m-esr52-fx_mark_as_shipped/build/venv/lib/python2.7/site-packages/shipitapi/shipitapi.py", line 61, in request > 01:42:41 FATAL - res.raise_for_status() > 01:42:41 FATAL - File "/builds/slave/rel-m-esr52-fx_mark_as_shipped/build/venv/lib/python2.7/site-packages/requests/models.py", line 909, in raise_for_status > 01:42:41 FATAL - raise HTTPError(http_error_msg, response=self) > 01:42:41 FATAL - HTTPError: 401 Client Error: Authorization Required for url: https://ship-it.mozilla.org/csrf_token > 01:42:41 FATAL - Running post_fatal callback... > 01:42:41 FATAL - Exiting -1
See Also: → 1407174
Summary: publish to balrog and mark release as shipped builders failed for 52.3.0esr → Cannot mark 52esr as shipped: HTTPError: 401 Client Error: Authorization Required for url: https://ship-it.mozilla.org/csrf_token
Rail, would you mind manually marking 52.4.1 as shipped?
Flags: needinfo?(rail)
:jcristau reminded me we can just click on a button on the web interface, which I just did.
Flags: needinfo?(rail)
(In reply to Johan Lorenzo [:jlorenzo] from comment #5) > :jcristau reminded me we can just click on a button on the web interface, > which I just did. thanks Johan, I'm just going through the list of stuff with my name along side it now. Is there anything else to do here? I seem to recall that we will have to do this for the next esr too? Perhaps I am to put that in releasewarrior notes.
You need to log in before you can comment on or make changes to this bug.