Blobber errors in all runs: "ValueError: No JSON object could be decoded" and "simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: emorley, Assigned: mtabara)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
We're seeing blobber errors even in otherwise green runs:

https://tbpl.mozilla.org/php/getParsedLog.php?id=48384828&tree=Mozilla-Inbound
{
08:27:27     INFO - Preparing to upload files from /builds/slave/test/build/blobber_upload_dir.
08:27:27     INFO - Files from /builds/slave/test/build/blobber_upload_dir are to be uploaded with <mozilla-inbound> branch at the following location(s): https://blobupload.elasticbeanstalk.com
08:27:27     INFO - Running command: ['/builds/slave/test/build/venv/bin/python', '/builds/slave/test/build/venv/bin/blobberc.py', '-u', 'https://blobupload.elasticbeanstalk.com', '-a', '/builds/slave/test/oauth.txt', '-b', 'mozilla-inbound', '-d', '/builds/slave/test/build/blobber_upload_dir', '--output-manifest-url', '/builds/slave/test/build/blobber_manifest.txt']
08:27:27     INFO - Copy/paste: /builds/slave/test/build/venv/bin/python /builds/slave/test/build/venv/bin/blobberc.py -u https://blobupload.elasticbeanstalk.com -a /builds/slave/test/oauth.txt -b mozilla-inbound -d /builds/slave/test/build/blobber_upload_dir --output-manifest-url /builds/slave/test/build/blobber_manifest.txt
08:27:28     INFO -  Traceback (most recent call last):
08:27:28     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 273, in <module>
08:27:28     INFO -      main()
08:27:28     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 256, in main
08:27:28     INFO -      filetype_whitelist = get_server_whitelist(args['--url'])
08:27:28     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 69, in get_server_whitelist
08:27:28     INFO -      return set(response.json().get('whitelist', []))
08:27:28     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/models.py", line 651, in json
08:27:28     INFO -      return json.loads(self.text or self.content, **kwargs)
08:27:28     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/simplejson/__init__.py", line 488, in loads
08:27:28     INFO -      return _default_decoder.decode(s)
08:27:28     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
08:27:28     INFO -      obj, end = self.raw_decode(s)
08:27:28     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/simplejson/decoder.py", line 389, in raw_decode
08:27:28     INFO -      return self.scan_once(s, idx=_w(s, idx).end())
08:27:28     INFO -  simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
}

https://tbpl.mozilla.org/php/getParsedLog.php?id=48353025&tree=Mozilla-Central
{
00:44:56     INFO - Preparing to upload files from /builds/slave/test/build/blobber_upload_dir.
00:44:56     INFO - Files from /builds/slave/test/build/blobber_upload_dir are to be uploaded with <mozilla-central> branch at the following location(s): https://blobupload.elasticbeanstalk.com
00:44:56     INFO - Running command: ['/builds/slave/test/build/venv/bin/python', '/builds/slave/test/build/venv/bin/blobberc.py', '-u', 'https://blobupload.elasticbeanstalk.com', '-a', '/builds/slave/test/oauth.txt', '-b', 'mozilla-central', '-d', '/builds/slave/test/build/blobber_upload_dir', '--output-manifest-url', '/builds/slave/test/build/blobber_manifest.txt']
00:44:56     INFO - Copy/paste: /builds/slave/test/build/venv/bin/python /builds/slave/test/build/venv/bin/blobberc.py -u https://blobupload.elasticbeanstalk.com -a /builds/slave/test/oauth.txt -b mozilla-central -d /builds/slave/test/build/blobber_upload_dir --output-manifest-url /builds/slave/test/build/blobber_manifest.txt
00:44:57     INFO -  Traceback (most recent call last):
00:44:57     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 273, in <module>
00:44:57     INFO -      main()
00:44:57     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 256, in main
00:44:57     INFO -      filetype_whitelist = get_server_whitelist(args['--url'])
00:44:57     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 69, in get_server_whitelist
00:44:57     INFO -      return set(response.json().get('whitelist', []))
00:44:57     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/models.py", line 651, in json
00:44:57     INFO -      return json.loads(self.text or self.content, **kwargs)
00:44:57     INFO -    File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
00:44:57     INFO -      return _default_decoder.decode(s)
00:44:57     INFO -    File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
00:44:57     INFO -      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
00:44:57     INFO -    File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
00:44:57     INFO -      raise ValueError("No JSON object could be decoded")
00:44:57     INFO -  ValueError: No JSON object could be decoded
}

The latter seems like bug 1066187 fallout, since it's a marionette-webapi. However the former is on desktop linux mochitest-3.
(Assignee)

Comment 1

3 years ago
It seems that we've been having our blobber elastic beanstalk servers down for some reason. Rebuild environment, seems that we have all green once again on our side. I've retriggered the job. Will be back with details.
(Assignee)

Updated

3 years ago
Assignee: nobody → mtabara
(Assignee)

Comment 2

3 years ago
Retriggered the job - https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/build/51531308 and got the files successfully uploaded with blobber: http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/44646de547dca3cf93941ec023ddb7006817d2ebb1ddfc54832f317aa36300d981f052ca82047cce838013c2dbab46b415e94ebe45555800722894507e27dce7

Should we wait for other retriggered jobs to go green or can we close this?
(Reporter)

Comment 3

3 years ago
The blobber failures weren't making the job fail, they appeared in all jobs - even green (per the summary). As such, we can just close this out now. Thank you for sorting this :-)
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.