1078248 - android single locale nightly builds need to retry balrog submissions

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

I noticed this in bug 1077381 - we have Balrog submissions failing sometimes with:
04:25:09     INFO - Calling Balrog submission script
04:25:09     INFO - retry: Calling <bound method MobileSingleLocale.run_command of <__main__.MobileSingleLocale object at 0x296a590>> with args: (['python', '/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-cen-and-l10n_1-0000000000000/balrog_props.json', '--api-root', 'https://aus4-admin.mozilla.org', '--username', 'ffxbld', '-t', 'nightly', '--credentials-file', '/builds/slave/m-cen-and-l10n_1-0000000000000/oauth.txt', '--verbose'],), kwargs: {}, attempt #1
04:25:09     INFO - Running command: ['python', '/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-cen-and-l10n_1-0000000000000/balrog_props.json', '--api-root', 'https://aus4-admin.mozilla.org', '--username', 'ffxbld', '-t', 'nightly', '--credentials-file', '/builds/slave/m-cen-and-l10n_1-0000000000000/oauth.txt', '--verbose']
04:25:09     INFO - Copy/paste: python /builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/balrog-submitter.py --build-properties /builds/slave/m-cen-and-l10n_1-0000000000000/balrog_props.json --api-root https://aus4-admin.mozilla.org --username ffxbld -t nightly --credentials-file /builds/slave/m-cen-and-l10n_1-0000000000000/oauth.txt --verbose
04:25:09     INFO -  Balrog request to https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-20141005030205
04:25:09     INFO -  Data sent: None
04:25:09     INFO -  Starting new HTTPS connection (1): aus4-admin.mozilla.org
04:25:10     INFO -  "HEAD /releases/Fennec-mozilla-central-nightly-20141005030205 HTTP/1.1" 200 0
04:25:10     INFO -  Balrog request to https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-20141005030205/builds/Android_arm-eabi-gcc3/et
04:25:10     INFO -  Data sent: {'product': u'Fennec', 'hashFunction': u'sha512', 'alias': 'null', 'schema_version': 3, 'data_version': '21', 'copyTo': '["Fennec-mozilla-central-nightly-latest"]', 'version': u'35.0a1', 'data': '{"buildID": "20141005030205", "platformVersion": "35.0a1", "displayVersion": "35.0a1", "appVersion": "35.0a1", "completes": [{"fileUrl": "http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014/10/2014-10-05-03-02-05-mozilla-central-android-l10n/fennec-35.0a1.et.android-arm.apk", "hashValue": "3be94be0ca628a024b7177120beed62ec599e3c91ffa980eea8cc30e5f0b28ed32cf7a22d000aa234e1d352761719da46c9ec902f5274526cb3c7cc2c2b33c1e", "from": "*", "filesize": 39089542}]}'}
04:25:10     INFO -  Starting new HTTPS connection (2): aus4-admin.mozilla.org
04:25:11     INFO -  "PUT /releases/Fennec-mozilla-central-nightly-20141005030205/builds/Android_arm-eabi-gcc3/et HTTP/1.1" 400 98
04:25:11     INFO -  Caught HTTPError: Couldn't update release: Failed to update row, old_data_version doesn't match current data_version
04:25:11     INFO -  Traceback (most recent call last):
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/balrog-submitter.py", line 78, in <module>
04:25:11     INFO -      extVersion, isOSUpdate=isOSUpdate, **updateKwargs)
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 246, in run
04:25:11     INFO -      return NightlySubmitterBase.run(self, *args, schemaVersion=3, **kwargs)
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 212, in run
04:25:11     INFO -      schemaVersion=schemaVersion)
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 162, in update_build
04:25:11     INFO -      url_template_vars=url_template_vars)
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 103, in request
04:25:11     INFO -      return self.do_request(url, data, method, url_template_vars)
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 118, in do_request
04:25:11     INFO -      headers=headers)
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
04:25:11     INFO -      r.send(prefetch=prefetch)
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
04:25:11     INFO -      self.response.raise_for_status()
04:25:11     INFO -    File "/builds/slave/m-cen-and-l10n_1-0000000000000/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
04:25:11     INFO -      raise http_error
04:25:11     INFO -  requests.exceptions.HTTPError: 400 Client Error


Looks to me like the submission is failing due to losing a race, but it's not retrying.

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1019724dummy.patch

I added this patch my staging env and my dummy build was triggered after all the l10n repacks completed.  I thought the email would magically get sent but I can't see that it happened here

http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/release-mozilla-beta-android_repack_complete

So I'm not sure what I'm missing, help appreciated

Comment 2

[Kim Moir [:kmoir] ET]

Whoops previous comment was on the wrong android bug, let me put this on the correct one.

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: bug1078248.patch

I think this is because retry attempts has a default of None

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 8501035 [details] [diff] [review]
bug1078248.patch

Review of attachment 8501035 [details] [diff] [review]:
-----------------------------------------------------------------

Huh, that's a weird default.

Comment 5

[Kim Moir [:kmoir] ET]

Closing, merged