1151459 - Intermittent nightly failures with "requests.exceptions.HTTPError: 400 Client Error" from "old_data_version doesn't match current data_version"

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Phil Ringnalda (:philor)]

https://treeherder.mozilla.org/logviewer.html#?job_id=742195&repo=mozilla-aurora

(aurora Linux en-US)

03:08:41 INFO - Calling Balrog submission script
03:08:41 INFO - retry: Calling run_command with args: (['python', '/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-aurora-lx-ntly-0000000000000/balrog_props.json', '-t', 'nightly', '--credentials-file', '/builds/slave/m-aurora-lx-ntly-0000000000000/oauth.txt', '--verbose', '--api-root', 'https://aus4-admin.mozilla.org/api', '--username', 'ffxbld', '--url-replacement', 'http://ftp.mozilla.org/pub/mozilla.org,http://download.cdn.mozilla.net/pub/mozilla.org'],), kwargs: {}, attempt #1
03:08:41 INFO - Running command: ['python', '/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-aurora-lx-ntly-0000000000000/balrog_props.json', '-t', 'nightly', '--credentials-file', '/builds/slave/m-aurora-lx-ntly-0000000000000/oauth.txt', '--verbose', '--api-root', 'https://aus4-admin.mozilla.org/api', '--username', 'ffxbld', '--url-replacement', 'http://ftp.mozilla.org/pub/mozilla.org,http://download.cdn.mozilla.net/pub/mozilla.org']
03:08:41 INFO - Copy/paste: python /builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/balrog-submitter.py --build-properties /builds/slave/m-aurora-lx-ntly-0000000000000/balrog_props.json -t nightly --credentials-file /builds/slave/m-aurora-lx-ntly-0000000000000/oauth.txt --verbose --api-root https://aus4-admin.mozilla.org/api --username ffxbld --url-replacement http://ftp.mozilla.org/pub/mozilla.org,http://download.cdn.mozilla.net/pub/mozilla.org
03:08:41 INFO - Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/04/2015-04-05-00-40-04-mozilla-aurora/firefox-39.0a2.en-US.linux-i686.complete.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-05-00-40-04-mozilla-aurora/firefox-39.0a2.en-US.linux-i686.complete.mar
03:08:41 INFO - Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/04/2015-04-05-00-40-04-mozilla-aurora/firefox-39.0a2.en-US.linux-i686.partial.20150404004006-20150405004004.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-05-00-40-04-mozilla-aurora/firefox-39.0a2.en-US.linux-i686.partial.20150404004006-20150405004004.mar
03:08:41 INFO - Balrog request to https://aus4-admin.mozilla.org/api/releases/Firefox-mozilla-aurora-nightly-20150405004004
03:08:41 INFO - Data sent: None
03:08:41 INFO - Starting new HTTPS connection (1): aus4-admin.mozilla.org
03:08:42 INFO - "HEAD /api/releases/Firefox-mozilla-aurora-nightly-20150405004004 HTTP/1.1" 200 0
03:08:42 INFO - Balrog request to https://aus4-admin.mozilla.org/api/releases/Firefox-mozilla-aurora-nightly-20150405004004/builds/Linux_x86-gcc3/en-US
03:08:42 INFO - Data sent: {'product': u'Firefox', 'hashFunction': u'sha512', 'alias': 'null', 'schema_version': 4, 'data_version': '48', 'copyTo': '["Firefox-mozilla-aurora-nightly-latest"]', 'version': u'39.0a2', 'data': '{"platformVersion": "39.0a2", "partials": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-05-00-40-04-mozilla-aurora/firefox-39.0a2.en-US.linux-i686.partial.20150404004006-20150405004004.mar", "hashValue": "2bd2e792ae1382dc76206152fc876ee970023fe9600ef963846cc66629a0d878a5f0e46507a3a494018b3d940f9688305990978451730333fb64e6fa598de505", "from": "Firefox-mozilla-aurora-nightly-20150404004006", "filesize": 5330372}], "completes": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-05-00-40-04-mozilla-aurora/firefox-39.0a2.en-US.linux-i686.complete.mar", "hashValue": "f49233f526a01d8ecfa27c518cef8b310f7951e4af32d1047c3c73668754618b4e977b77bdd6cbb4040f43c0e2cd29c434a377cf52b666b8acf3e48ab9c4b2e5", "from": "*", "filesize": 53807309}], "buildID": "20150405004004", "displayVersion": "39.0a2", "appVersion": "39.0a2"}'}
03:08:42 INFO - Starting new HTTPS connection (2): aus4-admin.mozilla.org
03:08:42 INFO - "PUT /api/releases/Firefox-mozilla-aurora-nightly-20150405004004/builds/Linux_x86-gcc3/en-US HTTP/1.1" 400 98
03:08:42 INFO - Caught HTTPError: Couldn't update release: Failed to update row, old_data_version doesn't match current data_version
03:08:42 INFO - Traceback (most recent call last):
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/balrog-submitter.py", line 86, in <module>
03:08:42 INFO - extVersion, isOSUpdate=isOSUpdate, **updateKwargs)
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 352, in run
03:08:42 INFO - return NightlySubmitterBase.run(self, *args, schemaVersion=4, **kwargs)
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 304, in run
03:08:42 INFO - schemaVersion=schemaVersion)
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 163, in update_build
03:08:42 INFO - url_template_vars=url_template_vars)
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 103, in request
03:08:42 INFO - return self.do_request(url, data, method, url_template_vars)
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 119, in do_request
03:08:42 INFO - headers=headers)
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
03:08:42 INFO - r.send(prefetch=prefetch)
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
03:08:42 INFO - self.response.raise_for_status()
03:08:42 INFO - File "/builds/slave/m-aurora-lx-ntly-0000000000000/build/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
03:08:42 INFO - raise http_error
03:08:42 INFO - requests.exceptions.HTTPError: 400 Client Error
03:08:42 ERROR - Return code: 1

https://treeherder.mozilla.org/logviewer.html#?job_id=1290536&repo=mozilla-central

(m-c OS X l10n)

07:00:29 INFO - Running command: ['python', '/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/balrog-submitter.py', '--build-properties', '/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/balrog_props.json', '-t', 'nightly', '--credentials-file', '/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/oauth.txt', '--verbose', '--api-root', 'https://aus4-admin.mozilla.org/api', '--username', 'ffxbld', '--url-replacement', 'http://ftp.mozilla.org/pub/mozilla.org,http://download.cdn.mozilla.net/pub/mozilla.org']
07:00:29 INFO - Copy/paste: python /builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/balrog-submitter.py --build-properties /builds/slave/m-cen-m64-ntly-l10n-2_3-000000/balrog_props.json -t nightly --credentials-file /builds/slave/m-cen-m64-ntly-l10n-2_3-000000/oauth.txt --verbose --api-root https://aus4-admin.mozilla.org/api --username ffxbld --url-replacement http://ftp.mozilla.org/pub/mozilla.org,http://download.cdn.mozilla.net/pub/mozilla.org
07:00:29 INFO - Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/04/2015-04-06-03-02-04-mozilla-central-l10n/firefox-40.0a1.nb-NO.mac.complete.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-06-03-02-04-mozilla-central-l10n/firefox-40.0a1.nb-NO.mac.complete.mar
07:00:29 INFO - Replacing http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/04/2015-04-06-03-02-04-mozilla-central-l10n/firefox-40.0a1.nb-NO.mac.partial.20150406030204-20150406051805.mar with http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-06-03-02-04-mozilla-central-l10n/firefox-40.0a1.nb-NO.mac.partial.20150406030204-20150406051805.mar
07:00:29 INFO - Balrog request to https://aus4-admin.mozilla.org/api/releases/Firefox-mozilla-central-nightly-20150406030204
07:00:29 INFO - Data sent: None
07:00:29 INFO - Starting new HTTPS connection (1): aus4-admin.mozilla.org
07:00:29 INFO - "HEAD /api/releases/Firefox-mozilla-central-nightly-20150406030204 HTTP/1.1" 200 0
07:00:29 INFO - Balrog request to https://aus4-admin.mozilla.org/api/releases/Firefox-mozilla-central-nightly-20150406030204/builds/Darwin_x86_64-gcc3-u-i386-x86_64/nb-NO
07:00:29 INFO - Data sent: {'product': u'Firefox', 'hashFunction': u'sha512', 'alias': '["Darwin_x86-gcc3-u-i386-x86_64", "Darwin_x86-gcc3", "Darwin_x86_64-gcc3"]', 'schema_version': 4, 'data_version': '97', 'copyTo': '["Firefox-mozilla-central-nightly-latest"]', 'version': u'40.0a1', 'data': '{"platformVersion": "40.0a1", "partials": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-06-03-02-04-mozilla-central-l10n/firefox-40.0a1.nb-NO.mac.partial.20150406030204-20150406051805.mar", "hashValue": "1826c2a59ae26d3dea294e118a371ed3854b9185052ced8161a4fcc59f41c9b7338507d30269bc69dc48a3dbc3360fe973f7fd64add3c7c4355f058802b8f7a2", "from": "Firefox-mozilla-central-nightly-20150406030204", "filesize": 400512}], "completes": [{"fileUrl": "http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/2015/04/2015-04-06-03-02-04-mozilla-central-l10n/firefox-40.0a1.nb-NO.mac.complete.mar", "hashValue": "a07d844cba1644f72cd554990e2d5d0f88a9a359c241312c09107349167c4f2fe8a99d91a016c645c647e88296cff5f1c4627c18926ed4f58932ae72ec857de1", "from": "*", "filesize": 96585520}], "buildID": "20150406030204", "displayVersion": "40.0a1", "appVersion": "40.0a1"}'}
07:00:29 INFO - Starting new HTTPS connection (2): aus4-admin.mozilla.org
07:00:29 INFO - "PUT /api/releases/Firefox-mozilla-central-nightly-20150406030204/builds/Darwin_x86_64-gcc3-u-i386-x86_64/nb-NO HTTP/1.1" 400 98
07:00:29 INFO - Caught HTTPError: Couldn't update release: Failed to update row, old_data_version doesn't match current data_version
07:00:29 INFO - Traceback (most recent call last):
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/balrog-submitter.py", line 86, in <module>
07:00:29 INFO - extVersion, isOSUpdate=isOSUpdate, **updateKwargs)
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 352, in run
07:00:29 INFO - return NightlySubmitterBase.run(self, *args, schemaVersion=4, **kwargs)
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/cli.py", line 304, in run
07:00:29 INFO - schemaVersion=schemaVersion)
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 163, in update_build
07:00:29 INFO - url_template_vars=url_template_vars)
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 103, in request
07:00:29 INFO - return self.do_request(url, data, method, url_template_vars)
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/balrog/submitter/api.py", line 119, in do_request
07:00:29 INFO - headers=headers)
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
07:00:29 INFO - r.send(prefetch=prefetch)
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
07:00:29 INFO - self.response.raise_for_status()
07:00:29 INFO - File "/builds/slave/m-cen-m64-ntly-l10n-2_3-000000/build/tools/scripts/updates/../../lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
07:00:29 INFO - raise http_error
07:00:29 INFO - requests.exceptions.HTTPError: 400 Client Error
07:00:30 ERROR - Return code: 1

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

old_data_version can be wrong if there is a race between two different jobs submitting new info at the same time (ie data_version is just a counter for how many times the release blob has been modified).

The retry at http://hg.mozilla.org/build/mozharness/file/d80001d9daf9/mozharness/mozilla/updates/balrog.py#l78 is supposed to catch this case but doesn't seem to be working.

Comment 2

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1322470
repository: mozilla-central
start_time: 2015-04-14T05:19:33
who: rvandermeulen[at]mozilla[dot]com
machine: bld-linux64-spot-009
buildname: Firefox mozilla-central linux64 l10n nightly-1
revision: 7f343964210b

requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(eo)
failure: submit_repack_to_balrog(eo)

Comment 3

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1318693
repository: mozilla-central
start_time: 2015-04-13T05:28:55
who: rvandermeulen[at]mozilla[dot]com
machine: bld-linux64-spot-1044
buildname: Firefox mozilla-central linux l10n nightly-2
revision: 2c9708e6b54d

requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(hu)
failure: submit_repack_to_balrog(hu)

Comment 4

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1322481
repository: mozilla-central
start_time: 2015-04-14T05:27:41
who: rvandermeulen[at]mozilla[dot]com
machine: bld-linux64-spot-216
buildname: Firefox mozilla-central linux l10n nightly-3
revision: 7f343964210b

requests.exceptions.HTTPError: 400 Client Error
Return code: 1
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(pl)
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(pt-BR)
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(pt-PT)
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(ru)
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(sk)
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(sl)
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(sv-SE)
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
requests.exceptions.HTTPError: 400 Client Error
Return code: 1
failure: submit_repack_to_balrog(th)
failure: submit_repack_to_balrog(pl)
failure: submit_repack_to_balrog(pt-BR)
failure: submit_repack_to_balrog(pt-PT)
failure: submit_repack_to_balrog(ru)
failure: submit_repack_to_balrog(sk)
failure: submit_repack_to_balrog(sl)
failure: submit_repack_to_balrog(sv-SE)
failure: submit_repack_to_balrog(th)

Comment 5

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1535847
repository: mozilla-central
start_time: 2015-05-21T03:02:42
who: tomcat[at]mozilla[dot]com
machine: bld-lion-r5-064
buildname: OS X 10.7 mozilla-central nightly
revision: b9424d63fe35

requests.exceptions.HTTPError: 400 Client Error
Return code: 1

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

The first hunk of http://hg.mozilla.org/build/mozharness/rev/bcf7da255f66 should fix this up. Please reopen if that's not the case.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We still have lots of bustage due to this problem since ages across all platforms:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&filter-searchStr=l10n%20nightly&fromchange=96c8de96e3ce&exclusion_profile=false

So the mentioned fix didn't solve it. Reopening accordingly.

Comment 9

[Rail Aliiev [:rail]]

This one is a bit different. 

it is about a similar error (well, exactly the same), but in a different context - it wasn't retrying. The current issue is that we retry, fail, sleep, retry, sleep moar, retry, .... give up

Comment 10

[Rail Aliiev [:rail]]

bug 1223872 to address this issue in comment #8

Comment 11

[Intermittent Failures Robot]

61 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 61

Platform breakdown:
* linux32: 23
* windows8-32: 14
* windows8-64: 12
* linux64: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1151459&startday=2015-11-09&endday=2015-11-15&tree=all

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Please disregard comment 11 from OrangeFactor Robot. Those failures are related to bug 1223872.