1413910 - fix issues during staging release of 57.0 in releng configs

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Description

[Kim Moir [:kmoir] ET]

So we can run a staging release for release

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1413910.patch

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8924543 [details] [diff] [review]
bug1413910.patch

Review of attachment 8924543 [details] [diff] [review]:
-----------------------------------------------------------------

Ship it!

Comment 3

[Kim Moir [:kmoir] ET]

Comment on attachment 8924543 [details] [diff] [review]
bug1413910.patch

and merged to prod

Comment 4

[Kim Moir [:kmoir] ET]

Attachment: bug1413910-2.patch

Funsize jobs are failing like this
https://tools.taskcluster.net/groups/DPaW3lf_QhqjEQspnmzjOA/tasks/EDDyGjqyRPm53O79b3gMNw/runs/0/logs/public%2Flogs%2Flive_backing.log

2017-11-02 15:58:32,269 - DEBUG - "PUT /api/releases/Firefox-57.0-build2/builds/Linux_x86-gcc3/en-US HTTP/1.1" 403 227
2017-11-02 15:58:32,269 - ERROR - Caught HTTPError: {"status": 403, "exception": "Permission denied to perform the request. balrog-stage-ffxbld is not allowed to create releases for product Firefox", "type": "about:blank", "detail": "PermissionDeniedError", "title": "Forbidden"}
2017-11-02 15:58:32,270 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-57.0-build2/builds/Linux_x86-gcc3/en-US", "timestamp": 1509638312.269966, "method": "PUT", "elapsed_secs": 0.2405109405517578, "status_code": 403}
2017-11-02 15:58:32,270 - DEBUG - retry: Caught exception:

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8924652 [details] [diff] [review]
bug1413910-2.patch

Review of attachment 8924652 [details] [diff] [review]:
-----------------------------------------------------------------

I think you need to use https://balrog-admin.stage.mozaws.net/api instead of that IP.

Comment 6

[Kim Moir [:kmoir] ET]

Attachment: bug1413910-3.patch

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8924660 [details] [diff] [review]
bug1413910-3.patch

Review of attachment 8924660 [details] [diff] [review]:
-----------------------------------------------------------------

Ship it!

Comment 8

[Kim Moir [:kmoir] ET]

So there is a failure in my staging release

https://public.etherpad-mozilla.org/p/staging-releases-jamun

And I noticed that on line 42

testing/mozharness/configs/releases/dev_updates_firefox_release.py 

refers to mozJamun-branch-patcher2.cfg
 "release": {
            "version_regex": r"^\d+\.\d+(\.\d+)?$",
            "requires_mirrors": True,
            "patcher_config": "mozJamun-branch-patcher2.cfg"

which doesn't exist in tools, should this be added to the staging tools repo or is it not needed?  I'm not that familiar with patcher configs.

Comment 9

[Rail Aliiev [:rail]]

We need to (in the tools repo):
* hg copy release/patcher-configs/moz{Release,Jamun}-branch-patcher2.cfg 
* touch release/updates/jamun-firefox-{linux{,64},macosx64,win{32,64}}.cfg
* hg add && hg commit -m "Add jamun patcher and update verify configs"

Comment 10

[Rail Aliiev [:rail]]

... and push!

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Also, just a heads-up completion. IIRC, you can push the files to staging tools and rerun the updates builder job without the need of starting another follow-up staging release.  

Updates builder does two things: 
a) patcher-configs generation (updating the files + creating a hg tag) to be used by update verification builders (finds the files by the hg tag)
b) submit top-level information to Balrog

For your current staging release firefox 57.0 build3 the builder didn't make it to the hg tag creation logic yet because it failed to find the proper files as you said in comment 8. But for future, if you have trouble with update verification tasks and you need a fix in the patcher configs, you can always bump and overwrite the hg tag in the staging tools repo and rerun those tasks. It helps to avoid triggering new staging releases to debug update verification.

Comment 12

[Kim Moir [:kmoir] ET]

thanks, trying another staging build with those suggestions.

Comment 13

[Kim Moir [:kmoir] ET]

Attachment: bug1413910-4.patch for jamun

bump_configs was still failing in last night's staging build

https://tools.taskcluster.net/groups/Eggp1B7NTxusYxKu8vFd7w/tasks/JXihilq0QVe6xjesOdLy0w/details

I debugged it locally on my machine and it seemed the problem was that it iterated over the beta channels first and since they didn't match it failed. This is just a temporary fix to address the issue on jamun to run release staging builds

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 8925058 [details] [diff] [review]
bug1413910-4.patch for jamun

Sure, let's try this approach!

Comment 15

[Kim Moir [:kmoir] ET]

Comment on attachment 8924660 [details] [diff] [review]
bug1413910-3.patch

ut now all the funsize publish to balrog jobs are still failing, so investigating that 
3:54 PM so looking at previous staging releases for beta on jamun, they used http://balrog/api/releases/ instead of 
3:54 PM https://balrog-admin.stage.mozaws.net/api
3:54 PM so perhaps I didn't need this patch
3:54 PM https://bug1413910.bmoattachments.org/attachment.cgi?id=8924660
3:57 PM 
<bhearsum> hmmm, yeah
3:57 PM it uses the vpnproxy, so it has to go through http://balrog/api
3:57 PM 
<kmoir> Kim Moir right, that's what I was thinking
3:57 PM I reverted that patch

Comment 16

[Kim Moir [:kmoir] ET]

So the funsize balrog jobs failed again

https://tools.taskcluster.net/groups/X0D1-9IrRfOfa-7mPpzbaw/tasks/MFpSsqViQ4S1QtK3xhRuXw/runs/0/logs/public%2Flogs%2Flive_backing.log

2017-11-04 00:44:30,229 - DEBUG - Balrog request to http://balrog/api/csrf_token
2017-11-04 00:44:30,229 - DEBUG - Data sent: None
2017-11-04 00:44:30,449 - DEBUG - "HEAD /api/csrf_token HTTP/1.1" 200 0
2017-11-04 00:44:30,450 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/csrf_token", "timestamp": 1509756270.450419, "method": "HEAD", "elapsed_secs": 0.22090792655944824, "status_code": 200}
2017-11-04 00:44:30,450 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-57.0-build6/builds/Linux_x86-gcc3/ach
2017-11-04 00:44:30,450 - DEBUG - Data sent: {'data': '{"buildID": "20171103160923", "appVersion": "57.0", "displayVersion": "57.0", "partials": [{"hashValue": "83185e32813ecd04d44520c3e0a8d90b44ed3bebdf7aa3364200ce1e687beac12e4b69c3c62ab9ef02c2c1b0ad40c14a1202b165fd1a391b684c915d51848523", "from": "Firefox-56.0.1-build2", "filesize": 30363493}], "platformVersion": "57.0", "completes": [{"hashValue": "86d5a6a59aa462b009dfbc0aad063791275e800a0311d6c58187e1fa5af4828287c9d977913486c5cd3c1da1210e89fe4ab406e95ef4f7571c6e3df8e658eaa7", "from": "*", "filesize": 44952652}]}', 'product': 'Firefox', 'hashFunction': 'sha512', 'schema_version': 4}
2017-11-04 00:44:30,683 - DEBUG - "PUT /api/releases/Firefox-57.0-build6/builds/Linux_x86-gcc3/ach HTTP/1.1" 403 227
2017-11-04 00:44:30,683 - ERROR - Caught HTTPError: {"status": 403, "exception": "Permission denied to perform the request. balrog-stage-ffxbld is not allowed to create releases for product Firefox", "type": "about:blank", "detail": "PermissionDeniedError", "title": "Forbidden"}
2017-11-04 00:44:30,684 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-57.0-build6/builds/Linux_x86-gcc3/ach", "timestamp": 1509756270.68402, "method": "PUT", "elapsed_secs": 0.2332289218902588, "status_code": 403}
2017-11-04 00:44:30,684 - DEBUG - retry: Caught exception: 
Traceback (most recent call last):
  File "/home/worker/tools/lib/python/vendor/redo-1.4.1/redo/__init__.py", line 152, in retry
    return action(*args, **kwargs)
  File "/home/worker/bin/funsize-balrog-submitter.py", line 186, in <lambda>
    partialInfo=partial_info, completeInfo=complete_info),
  File "/home/worker/tools/lib/python/balrog/submitter/cli.py", line 491, in run
    return ReleaseSubmitterBase.run(self, *args, schemaVersion=4, **kwargs)
  File "/home/worker/tools/lib/python/balrog/submitter/cli.py", line 448, in run
    schemaVersion=schemaVersion)
  File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.4/balrogclient/api.py", line 224, in update_build
    return self.request(method='PUT', data=data)
  File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.4/balrogclient/api.py", line 111, in request
    return self.do_request(url, data, method)
  File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.4/balrogclient/api.py", line 130, in do_request
    req.raise_for_status()
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 825, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 403 Client Error: FORBIDDEN

Not sure if this is 
1) not pointing at the staging balrog instance
2) permissions issue with the staging balrog instance

will investigate

Comment 17

[Kim Moir [:kmoir] ET]

bhearsum: I ran another staging build last night to try to fix this issue, but it didn't work and I'm still seeing similar errors.  Are there special permissions that need to be set to allow balrog-stage-ffxbld is 
allowed to create new releases on the staging release channel?

see error above
{"status": 403, "exception": "Permission denied to perform the request. balrog-stage-ffxbld is not allowed to create releases for product Firefox", "type": "about:blank", "detail": "PermissionDeniedError", "title": "Forbidden"}

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Kim Moir [:kmoir] ET from comment #17)
> bhearsum: I ran another staging build last night to try to fix this issue,
> but it didn't work and I'm still seeing similar errors.  Are there special
> permissions that need to be set to allow balrog-stage-ffxbld is 
> allowed to create new releases on the staging release channel?
> 
> see error above
> {"status": 403, "exception": "Permission denied to perform the request.
> balrog-stage-ffxbld is not allowed to create releases for product Firefox",
> "type": "about:blank", "detail": "PermissionDeniedError", "title":
> "Forbidden"}

It looks like something went wrong with the permissions table during the database recreation. Relud, is there any chance we have logs for the database rebuild? Can you also manually run this SQL so I can re-add the rest of the stage permissions?
INSERT INTO permissions (username, permission, data_version) VALUES ("[e-mail]", "admin", 1);

Comment 19

[Daniel Thorn [:relud]]

emailed the logs to :bhearsum

Comment 20

[[e-mail] (:bhearsum)]

(In reply to Kim Moir [:kmoir] ET from comment #17)
> bhearsum: I ran another staging build last night to try to fix this issue,
> but it didn't work and I'm still seeing similar errors.  Are there special
> permissions that need to be set to allow balrog-stage-ffxbld is 
> allowed to create new releases on the staging release channel?
> 
> see error above
> {"status": 403, "exception": "Permission denied to perform the request.
> balrog-stage-ffxbld is not allowed to create releases for product Firefox",
> "type": "about:blank", "detail": "PermissionDeniedError", "title":
> "Forbidden"}

This should be resolved now.

Comment 21

[Kim Moir [:kmoir] ET]

Attachment: pr to update doc to reflect changes required running staging release builds

Comment 22

[Kim Moir [:kmoir] ET]

PR merged