Closed Bug 1298759 Opened 9 years ago Closed 9 years ago

Permanent ValueError: No JSON object could be decoded (download from https://blobupload.elasticbeanstalk.com fails)

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aryx, Unassigned)

References

Details

(Keywords: intermittent-failure)

This error message started to show up in the logs of some test types on Saturday and is permafailing since that time. https://treeherder.mozilla.org/logviewer.html#?job_id=11332402&repo=fx-team 13:46:12 INFO - Running post-run listener: _upload_blobber_files 13:46:12 INFO - Blob upload gear active. 13:46:12 INFO - Preparing to upload files from c:\slave\test\build\blobber_upload_dir. 13:46:12 INFO - Files from c:\slave\test\build\blobber_upload_dir are to be uploaded with <fx-team> branch at the following location(s): https://blobupload.elasticbeanstalk.com 13:46:12 INFO - Running command: ['c:\\slave\\test\\build\\venv\\Scripts\\python', 'c:\\slave\\test\\build\\venv\\Scripts\\blobberc.py', '-u', 'https://blobupload.elasticbeanstalk.com', '-a', 'c:\\slave\\test\\oauth.txt', '-b', 'fx-team', '-d', 'c:\\slave\\test\\build\\blobber_upload_dir', '--output-manifest', 'c:\\slave\\test\\build\\uploaded_files.json'] 13:46:12 INFO - Copy/paste: c:\slave\test\build\venv\Scripts\python c:\slave\test\build\venv\Scripts\blobberc.py -u https://blobupload.elasticbeanstalk.com -a c:\slave\test\oauth.txt -b fx-team -d c:\slave\test\build\blobber_upload_dir --output-manifest c:\slave\test\build\uploaded_files.json 13:46:13 INFO - Traceback (most recent call last): 13:46:13 INFO - File "c:\slave\test\build\venv\Scripts\blobberc.py", line 253, in <module> 13:46:13 INFO - main() 13:46:13 INFO - File "c:\slave\test\build\venv\Scripts\blobberc.py", line 235, in main 13:46:13 INFO - filetype_whitelist = get_server_whitelist(args['--url']) 13:46:13 INFO - File "c:\slave\test\build\venv\Scripts\blobberc.py", line 69, in get_server_whitelist 13:46:13 INFO - return set(response.json().get('whitelist', [])) 13:46:13 INFO - File "c:\slave\test\build\venv\lib\site-packages\requests\models.py", line 651, in json 13:46:13 INFO - return json.loads(self.text or self.content, **kwargs) 13:46:13 INFO - File "c:\mozilla-build\python27\Lib\json\__init__.py", line 326, in loads 13:46:13 INFO - return _default_decoder.decode(s) 13:46:13 INFO - File "c:\mozilla-build\python27\Lib\json\decoder.py", line 366, in decode 13:46:13 INFO - obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 13:46:13 INFO - File "c:\mozilla-build\python27\Lib\json\decoder.py", line 384, in raw_decode 13:46:13 INFO - raise ValueError("No JSON object could be decoded") 13:46:13 INFO - ValueError: No JSON object could be decoded 13:46:13 ERROR - Return code: 1
From checking this code it looks like we fail to parse the JSON data as retrieved from https://blobupload.elasticbeanstalk.com/blobs/whitelist. I cannot access this host, so it's hard to verify. We would need buildduty to have a look at this. Alin, could you have a look please?
Flags: needinfo?(aselagea)
As mentioned by :aryx in #buildduty, the error comes from here: https://github.com/mozilla/build-blobuploader/blob/master/blobberc.py#L62. I logged in on a t-w732-spot instance and tried to see what response I get for a request to 'https://blobupload.elasticbeanstalk.com/blobs/whitelist'. At a first look, the service seems unavailable: C:\slave\test\build\venv\Scripts>python.exe Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win 32 Type "help", "copyright", "credits" or "license" for more information. >>> import urlparse >>> import requests >>> from blobuploader import cert >>> hostname = "https://blobupload.elasticbeanstalk.com" >>> url = urlparse.urljoin(hostname, '/blobs/whitelist') >>> url 'https://blobupload.elasticbeanstalk.com/blobs/whitelist' >>> response = requests.get(url, verify=cert.where()) >>> response <Response [503]> >>> response.content '' >>> response.__dict__ {'cookies': <<class 'requests.cookies.RequestsCookieJar'>[]>, '_content': '', 'h eaders': CaseInsensitiveDict({'content-length': '0', 'connection': 'keep-alive'} ), 'url': u'https://blobupload.elasticbeanstalk.com/blobs/whitelist', 'status_co de': 503, '_content_consumed': True, 'encoding': None, 'request': <PreparedReque st [GET]>, 'connection': <requests.adapters.HTTPAdapter object at 0x0206B830>, ' elapsed': datetime.timedelta(0, 0, 690000), 'raw': <requests.packages.urllib3.re sponse.HTTPResponse object at 0x020734D0>, 'reason': 'Service Unavailable: Back- end server is at capacity', 'history': []} http://status.aws.amazon.com/ reports the service as operating normally though.
Flags: needinfo?(aselagea)
See Also: → 1117620
Rail rebuilded the application and it is working again now(thanks, Rail!).
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.