Windows 8 L10n Repack often failing on aurora with HTTPError: 400 Client Error: BAD REQUEST

RESOLVED WORKSFORME

Status

Release Engineering
General Automation
RESOLVED WORKSFORME
7 months ago
4 months ago

People

(Reporter: aryx, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

One failure for Thursday's nightly https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=fd8bdabb4813164e70b51d02b420e3659eb38536&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=l10n after bug bug 1344321 landed.

Saturday and Sunday's Nightly saw 5 out of 6 repacks for Win 8 x64 failing: https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=9df61b09aa1ce0b26486bb30c6ca63e89ac06100&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=l10n
Flags: needinfo?(bugspam.Callek)
Per one log this is more of a balrog publish failure, which we've seen before...

05:01:40     INFO -  "PUT /api/releases/Firefox-mozilla-aurora-nightly-latest/builds/WINNT_x86-msvc/af HTTP/1.1" 400 87
05:01:40     INFO -  Caught HTTPError: {"data": ["Failed to update row, old_data_version doesn't match current data_version"]}
05:01:40     INFO -  REQUEST STATS: {"url": "https://aus4-admin.mozilla.org/api/releases/Firefox-mozilla-aurora-nightly-latest/builds/WINNT_x86-msvc/af", "timestamp": 1489924900.63, "method": "PUT", "elapsed_secs": 4.871000051498413, "status_code": 400}
05:01:40     INFO -  retry: Caught exception:
05:01:40     INFO -  Traceback (most recent call last):
05:01:40     INFO -    File "c:\builds\moz2_slave\m-aurora-w32-l10n-ntly-1-00000\build\tools\lib\python\vendor\redo-1.4.1\redo\__init__.py", line 152, in retry
05:01:40     INFO -      return action(*args, **kwargs)
05:01:40     INFO -    File "c:\builds\moz2_slave\m-aurora-w32-l10n-ntly-1-00000\build\tools\lib\python\balrog\submitter\cli.py", line 338, in update_latest
05:01:40     INFO -      data_version=latest_data_version)
05:01:40     INFO -    File "c:\builds\moz2_slave\m-aurora-w32-l10n-ntly-1-00000\build\tools\lib\python\vendor\balrogclient-0.0.1\balrogclient\api.py", line 223, in update_build
05:01:40     INFO -      return self.request(method='PUT', data=data)
05:01:40     INFO -    File "c:\builds\moz2_slave\m-aurora-w32-l10n-ntly-1-00000\build\tools\lib\python\vendor\balrogclient-0.0.1\balrogclient\api.py", line 111, in request
05:01:40     INFO -      return self.do_request(url, data, method)
05:01:40     INFO -    File "c:\builds\moz2_slave\m-aurora-w32-l10n-ntly-1-00000\build\tools\lib\python\vendor\balrogclient-0.0.1\balrogclient\api.py", line 129, in do_request
05:01:40     INFO -      req.raise_for_status()
05:01:40     INFO -    File "c:\builds\moz2_slave\m-aurora-w32-l10n-ntly-1-00000\build\tools\scripts\updates\../../lib/python/vendor/requests-2.7.0\requests\models.py", line 851, in raise_for_status
05:01:40     INFO -      raise HTTPError(http_error_msg, response=self)
05:01:40     INFO -  HTTPError: 400 Client Error: BAD REQUEST
05:01:40     INFO -  retry: Giving up on <function update_latest at 0x02C80330>

MANY of the errors are INFO only due to RETRIES.

I think there is a bug to dupe to, but I can't find it... Ben do you know said bug?
Flags: needinfo?(bugspam.Callek) → needinfo?(bhearsum)
It's a bit surprising that this would happen all of a sudden - nothing has changed on the backend that would make it more likely to hit this. I wonder if we increased parallelism is funsize recently - that would certainly make it more likely to hit these races.
Flags: needinfo?(bhearsum)

Updated

4 months ago
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.