Closed
Bug 1303106
Opened 7 years ago
Closed 4 years ago
add "product" to release blobs
Categories
(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)
Release Engineering Graveyard
Applications: Balrog (backend)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: bhearsum, Assigned: vjoshi)
References
(Depends on 1 open bug)
Details
(Whiteboard: [lang=python][ready])
Attachments
(1 file)
1.08 KB,
patch
|
bhearsum
:
review+
rail
:
checked-in-
|
Details | Diff | Splinter Review |
After the work on bug 1041584 (moving domain whitelist parsing into blobs), it looks like we would benefit from putting "product" in the blob. Doing so would mean that we don't need to pass it in whenever we check a blob against the domain whitelist. We will need to continue to maintain the releases.product column, however, because it's necessary to enforce product-specific permissions on releases. This means that we'll need to ensure that the product column and the in-blob product stay in sync. We already do this for "name", so there should be a model for it.
Reporter | ||
Comment 1•7 years ago
|
||
Part 1 of this is in https://github.com/mozilla/balrog/pull/129, and now in production.
Assignee | ||
Comment 2•7 years ago
|
||
The first part involved changing the blob schemas, adding an optional "product" field. The second part will involve a script that'll consume the API, adding the product to all blobs. Then, we will make "product" a required field. Finally, we will make changes to the code, removing "product" from argument lists.
Reporter | ||
Comment 3•7 years ago
|
||
(In reply to Varun Joshi (:vjoshi) from comment #2) > The second part will involve a script that'll consume the API, adding the > product to all blobs. I ran Varun's script today. It updated the majority of the blobs, but some could not be updated. Some of them failed because of domains not in the whitelist, and some for other reasons. I'm tracking fixing these in bug 1309656.
Reporter | ||
Comment 4•7 years ago
|
||
Varun, it looks like the submission tools in https://github.com/mozilla/build-tools/blob/master/lib/python/balrog/submitter/cli.py probably need an update. While working on bug 1309656 that all the recently created nightlies are still missing "product".
Assignee | ||
Comment 5•7 years ago
|
||
I haven't been able to test this, but I figured we'd need to add the "data" field to ReleaseCreatorBase, NightlySubmitterBase and to ReleaseSubmitterBase. Also, do we need to change the product name to lowercase?
Flags: needinfo?(bhearsum)
Reporter | ||
Comment 6•7 years ago
|
||
Comment on attachment 8814394 [details] [diff] [review] Add "product" to submitted data Review of attachment 8814394 [details] [diff] [review]: ----------------------------------------------------------------- Definitely no need to lowercase it (we use a capitalized version everywhere within the context of Balrog). This should be fine, and safe. I'm going to wait until Monday to land it though.
Attachment #8814394 -
Flags: review+
Reporter | ||
Updated•7 years ago
|
Flags: needinfo?(bhearsum)
Reporter | ||
Updated•7 years ago
|
Attachment #8814394 -
Flags: checked-in+
Comment 7•7 years ago
|
||
I had to backout the change, it's causing HTTP 400, see https://tools.taskcluster.net/task-inspector/#1fEDLzCUTUGA_trDkLcCGA
Comment 8•7 years ago
|
||
Comment on attachment 8814394 [details] [diff] [review] Add "product" to submitted data https://hg.mozilla.org/build/tools/rev/1334d527cafb
Attachment #8814394 -
Flags: checked-in+ → checked-in-
Comment 9•7 years ago
|
||
I wonder if we should move the client code under the same repo and run integration tests as a part of CI...
Reporter | ||
Comment 10•7 years ago
|
||
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #9) > I wonder if we should move the client code under the same repo and run > integration tests as a part of CI... Yeah. That would give the code access to the jsonschemas, too. https://bugzilla.mozilla.org/show_bug.cgi?id=1320949 and https://bugzilla.mozilla.org/show_bug.cgi?id=1312868 are probably what we want.
Reporter | ||
Comment 11•7 years ago
|
||
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #7) > I had to backout the change, it's causing HTTP 400, see > https://tools.taskcluster.net/task-inspector/#1fEDLzCUTUGA_trDkLcCGA 2016-11-29 00:30:00,027 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-51.0b4-build1/builds/Linux_x86_64-gcc3/el 2016-11-29 00:30:00,027 - DEBUG - Data sent: {'data_version': '2', 'product': u'Firefox', 'hashFunction': 'sha512', 'data': '{"product": "Firefox", "buildID": "20161128075558", "appVersion": "51.0", "displayVersion": "51.0 Beta 4", "partials": [{"hashValue": "a1f0522310e983764a2acd94131f730177e06367591987d099e09f0692d1a2fece80d1bd6c0507c4fa5faba7bb8d0a754ca374a38243524f017efba42839633f", "from": "Firefox-51.0b3-build1", "filesize": 9956885}], "platformVersion": "51.0", "completes": [{"hashValue": "c384310f56fd8804958b5924de14d4d1231b442074eae3a217f8d46680a76824ffc181c33a4c090165b2475b540f0135c6f2c089b1b25c09f9cb62bf1fe8b655", "from": "*", "filesize": 56737352}]}', 'schema_version': '4'} 2016-11-29 00:30:00,176 - DEBUG - "PUT /api/releases/Firefox-51.0b4-build1/builds/Linux_x86_64-gcc3/el HTTP/1.1" 400 698 2016-11-29 00:30:00,177 - ERROR - Caught HTTPError: {"data": ["{'OS_FTP': 'linux-x86_64', 'locales': {u'el': {u'product': u'Firefox', u'buildID': u'20161128075558', u'appVersion': u'51.0', u'platformVersion': u'51.0', u'partials': [{u'hashValue': u'a1f0522310e983764a2acd94131f730177e06367591987d099e09f0692d1a2fece80d1bd6c0507c4fa5faba7bb8d0a754ca374a38243524f017efba42839633f', u'from': u'Firefox-51.0b3-build1', u'filesize': 9956885}], u'displayVersion': u'51.0 Beta 4', u'completes': [{u'hashValue': u'c384310f56fd8804958b5924de14d4d1231b442074eae3a217f8d46680a76824ffc181c33a4c090165b2475b540f0135c6f2c089b1b25c09f9cb62bf1fe8b655', u'from': u'*', u'filesize': 56737352}]}}, 'OS_BOUNCER': 'linux64'} is not valid under any of the given schemas"]} 2016-11-29 00:30:00,177 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-51.0b4-build1/builds/Linux_x86_64-gcc3/el", "timestamp": 1480379400.177141, "method": "PUT", "elapsed_secs": 0.14925289154052734, "status_code": 400} 2016-11-29 00:30:00,177 - DEBUG - retry: Caught exception: Traceback (most recent call last): File "/home/worker/tools/lib/python/vendor/redo-1.4.1/redo/__init__.py", line 152, in retry return action(*args, **kwargs) File "/home/worker/bin/funsize-balrog-submitter.py", line 167, in <lambda> partialInfo=partial_info, completeInfo=complete_info, File "/home/worker/tools/lib/python/balrog/submitter/cli.py", line 471, in run return ReleaseSubmitterBase.run(self, *args, schemaVersion=4, **kwargs) File "/home/worker/tools/lib/python/balrog/submitter/cli.py", line 428, in run schemaVersion=schemaVersion) File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.1/balrogclient/api.py", line 223, in update_build return self.request(method='PUT', data=data) File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.1/balrogclient/api.py", line 111, in request return self.do_request(url, data, method) File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.1/balrogclient/api.py", line 129, in do_request req.raise_for_status() File "/usr/lib/python2.7/dist-packages/requests/models.py", line 825, in raise_for_status raise HTTPError(http_error_msg, response=self) HTTPError: 400 Client Error: BAD REQUEST 2016-11-29 00:30:00,177 - INFO - retry: Giving up on <function <lambda> at 0x7f385b61fb90> Traceback (most recent call last): File "/home/worker/bin/funsize-balrog-submitter.py", line 207, in <module> main() File "/home/worker/bin/funsize-balrog-submitter.py", line 160, in main retry(lambda: submitter.run( File "/home/worker/tools/lib/python/vendor/redo-1.4.1/redo/__init__.py", line 152, in retry return action(*args, **kwargs) File "/home/worker/bin/funsize-balrog-submitter.py", line 167, in <lambda> partialInfo=partial_info, completeInfo=complete_info, File "/home/worker/tools/lib/python/balrog/submitter/cli.py", line 471, in run return ReleaseSubmitterBase.run(self, *args, schemaVersion=4, **kwargs) File "/home/worker/tools/lib/python/balrog/submitter/cli.py", line 428, in run schemaVersion=schemaVersion) File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.1/balrogclient/api.py", line 223, in update_build return self.request(method='PUT', data=data) File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.1/balrogclient/api.py", line 111, in request return self.do_request(url, data, method) File "/home/worker/tools/lib/python/vendor/balrogclient-0.0.1/balrogclient/api.py", line 129, in do_request req.raise_for_status() File "/usr/lib/python2.7/dist-packages/requests/models.py", line 825, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST
Comment hidden (Intermittent Failures Robot) |
Reporter | ||
Updated•6 years ago
|
Priority: -- → P3
Whiteboard: [lang=python][ready]
Reporter | ||
Comment 13•6 years ago
|
||
Varun, are you still planning to look at this?
Flags: needinfo?(varunj.1011)
Assignee | ||
Comment 14•6 years ago
|
||
Sorry for the delay! Yes, I will look at it soon.
Flags: needinfo?(varunj.1011)
Reporter | ||
Comment 15•4 years ago
|
||
This is as fixed as its going to be :)
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Updated•3 years ago
|
Product: Release Engineering → Release Engineering Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•