Closed Bug 1117810 Opened 11 years ago Closed 9 years ago

don't submit to balrog if a locale failed to upload

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: bhearsum, Assigned: kmoir)

Details

Eg: 04:43:14 INFO - "HEAD /releases/Fennec-mozilla-central-nightly-20150105030202 HTTP/1.1" 200 0 04:43:14 INFO - Balrog request to https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-20150105030202/builds/Android_arm-eabi-gcc3/fa 04:43:14 INFO - Data sent: {'product': u'Fennec', 'hashFunction': u'sha512', 'alias': 'null', 'schema_version': 4, 'data_version': '13', 'copyTo': '["Fennec-mozilla-central-nightly-latest"]', 'version': u'37.0a1', 'data': '{"buildID": "20150105030202", "platformVersion": "37.0a1", "displayVersion": "37.0a1", "appVersion": "37.0a1", "completes": [{"fileUrl": null, "hashValue": "b6ce097ae3bc38f6d7515993bdaebba6ce4b1c7b42223928493f507826b7ab2e93c1fd784ff6dacf8c265958648b3b1d24ddb02fe2539ca4dce51ac35dfd8e14", "from": "*", "filesize": 38375913}]}'} 04:43:14 INFO - Starting new HTTPS connection (2): aus4-admin.mozilla.org 04:43:14 INFO - "PUT /releases/Fennec-mozilla-central-nightly-20150105030202/builds/Android_arm-eabi-gcc3/fa HTTP/1.1" 500 541 04:43:14 INFO - Caught HTTPError: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 04:43:14 INFO - <html><head> 04:43:14 INFO - <title>500 Internal Server Error</title> 04:43:14 INFO - </head><body> 04:43:14 INFO - <h1>Internal Server Error</h1> 04:43:14 INFO - <p>The server encountered an internal error or 04:43:14 INFO - misconfiguration and was unable to complete 04:43:14 INFO - your request.</p> 04:43:14 INFO - <p>Please contact the server administrator, 04:43:14 INFO - webmaster@mozilla.com and inform them of the time the error occurred, 04:43:14 INFO - and anything you might have done that may have 04:43:14 INFO - caused the error.</p> 04:43:14 INFO - <p>More information about this error may be available 04:43:14 INFO - in the server error log.</p> 04:43:14 INFO - </body></html> 04:43:14 INFO - Traceback (most recent call last): 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/balrog-submitter.py", line 70, in <module> 04:43:14 INFO - extVersion, isOSUpdate=isOSUpdate, **updateKwargs) 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 317, in run 04:43:14 INFO - return NightlySubmitterBase.run(self, *args, schemaVersion=4, **kwargs) 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 272, in run 04:43:14 INFO - schemaVersion=schemaVersion) 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 163, in update_build 04:43:14 INFO - url_template_vars=url_template_vars) 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 103, in request 04:43:14 INFO - return self.do_request(url, data, method, url_template_vars) 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 119, in do_request 04:43:14 INFO - headers=headers) 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request 04:43:14 INFO - r.send(prefetch=prefetch) 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send 04:43:14 INFO - self.response.raise_for_status() 04:43:14 INFO - File "/builds/slave/m-cen-and-api-11-l10n_2-000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 816, in raise_for_status 04:43:14 INFO - raise http_error 04:43:14 INFO - requests.exceptions.HTTPError: 500 Server Error Note the "null" fileUrl. I'm not sure if this is related to the api level split, single locale changes, or both.
Flags: needinfo?(kmoir)
Flags: needinfo?(jlund)
I looked at this today. The fa apk wasn't uploaded on Jan 5 nightly but was on the 6th. Investigating why it failed on Monday night.
Assignee: nobody → kmoir
Flags: needinfo?(kmoir)
hm, interesting, based off the above snippet's buildid, I think you found that in this build: https://tbpl.mozilla.org/php/getParsedLog.php?id=55700194&full=1&branch=mozilla-central ? it looks like all locales repacked successfully but fa and gl locales failed to upload: 04:41:33 INFO - Command ['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_rsa', 'ffxbld@stage.mozilla.org', 'post_upload.py -b mozilla-central-android-api-11-l10n -p mobile -i 20150105030202 --release-to-latest --release-to-dated "/tmp/tmp.n7yKMHy9RS/" "/tmp/tmp.n7yKMHy9RS//fennec-37.0a1.gl.android-arm.apk" "/tmp/tmp.n7yKMHy9RS//gecko-unsigned-unaligned.apk" "/tmp/tmp.n7yKMHy9RS//fennec-37.0a1.gl.android-arm.checksums" "/tmp/tmp.n7yKMHy9RS//fennec-37.0a1.gl.android-arm.checksums.asc"'] returned non-zero exit code: 255 04:41:33 INFO - make: *** [upload] Error 2 so that's why fileUrl is none. IIUC, split apk shouldn't change the build behavior here. Maybe we shouldn't try to submit to balrog if a locale failed to upload? while looking at the other chunks I also discovered the following with chunk 5 which seemed odd: log: https://tbpl.mozilla.org/php/getParsedLog.php?id=55700731&full=1&branch=mozilla-central snippet: 04:52:12 INFO - Calling Balrog submission script 04:52:12 INFO - retry: Calling <bound method MobileSingleLocale.run_command of <__main__.MobileSingleLocale object at 0x1d5d710>> with args: (['python', '/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-cen-and-api-11-l10n_5-000000/balrog_props.json', '--api-root', 'https://aus4-admin.mozilla.org', '--username', 'ffxbld', '-t', 'nightly', '--credentials-file', '/builds/slave/m-cen-and-api-11-l10n_5-000000/oauth.txt', '--verbose'],), kwargs: {}, attempt #1 04:52:12 INFO - Running command: ['python', '/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-cen-and-api-11-l10n_5-000000/balrog_props.json', '--api-root', 'https://aus4-admin.mozilla.org', '--username', 'ffxbld', '-t', 'nightly', '--credentials-file', '/builds/slave/m-cen-and-api-11-l10n_5-000000/oauth.txt', '--verbose'] 04:52:12 INFO - Copy/paste: python /builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/balrog-submitter.py --build-properties /builds/slave/m-cen-and-api-11-l10n_5-000000/balrog_props.json --api-root https://aus4-admin.mozilla.org --username ffxbld -t nightly --credentials-file /builds/slave/m-cen-and-api-11-l10n_5-000000/oauth.txt --verbose 04:52:12 INFO - Balrog request to https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-20150105030202 04:52:12 INFO - Data sent: None 04:52:12 INFO - Starting new HTTPS connection (1): aus4-admin.mozilla.org 04:52:13 INFO - "HEAD /releases/Fennec-mozilla-central-nightly-20150105030202 HTTP/1.1" 200 0 04:52:13 INFO - Balrog request to https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-20150105030202/builds/Android_arm-eabi-gcc3/vi 04:52:13 INFO - Data sent: {'product': u'Fennec', 'hashFunction': u'sha512', 'alias': 'null', 'schema_version': 4, 'data_version': '35', 'copyTo': '["Fennec-mozilla-central-nightly-latest"]', 'version': u'37.0a1', 'data': '{"buildID": "20150105030202", "platformVersion": "37.0a1", "displayVersion": "37.0a1", "appVersion": "37.0a1", "completes": [{"fileUrl": "http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/01/2015-01-05-03-02-02-mozilla-central-android-api-11-l10n/fennec-37.0a1.vi.android-arm.apk", "hashValue": "786876596367427cf4f1b5e9b909ed66bebe4b7d051abb1cb13d62453aed5540618894807179f7d28f66118420482e61453911f9b6a303efc2b6cfbba6f7f9a9", "from": "*", "filesize": 38364629}]}'} 04:52:13 INFO - Starting new HTTPS connection (2): aus4-admin.mozilla.org 04:52:13 INFO - "PUT /releases/Fennec-mozilla-central-nightly-20150105030202/builds/Android_arm-eabi-gcc3/vi HTTP/1.1" 400 98 04:52:13 INFO - Caught HTTPError: Couldn't update release: Failed to update row, old_data_version doesn't match current data_version 04:52:13 INFO - Traceback (most recent call last): 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/balrog-submitter.py", line 70, in <module> 04:52:13 INFO - extVersion, isOSUpdate=isOSUpdate, **updateKwargs) 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 317, in run 04:52:13 INFO - return NightlySubmitterBase.run(self, *args, schemaVersion=4, **kwargs) 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 272, in run 04:52:13 INFO - schemaVersion=schemaVersion) 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 163, in update_build 04:52:13 INFO - url_template_vars=url_template_vars) 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 103, in request 04:52:13 INFO - return self.do_request(url, data, method, url_template_vars) 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 119, in do_request 04:52:13 INFO - headers=headers) 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request 04:52:13 INFO - r.send(prefetch=prefetch) 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send 04:52:13 INFO - self.response.raise_for_status() 04:52:13 INFO - File "/builds/slave/m-cen-and-api-11-l10n_5-000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status 04:52:13 INFO - raise http_error 04:52:13 INFO - requests.exceptions.HTTPError: 400 Client Error 04:52:13 ERROR - Return code: 1
Flags: needinfo?(jlund)
Summary: single locale repacks sometimes try to submit bad data to balrog → don't submit to balrog if a locale failed to upload
I'm marking reso/incomplete since I haven't heard of *any* reports of this happening since initial bug filing -- and a lot of infra/code has changed since then. If there is anything since 2016 in terms of failures I'm happy to re-look given a log.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → INCOMPLETE
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.