Closed Bug 958056 Opened 6 years ago Closed 6 years ago

Migrate blobber server to python 2.7 Elastic Beanstalk

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ahal, Assigned: rail)

Details

(Whiteboard: [blobber])

Across all branches blobber is hitting a critical error. There is a traceback on Ash:
https://tbpl.mozilla.org/php/getParsedLog.php?id=32046578&tree=Ash&full=1#error0

14:32:41     INFO -  (blobuploader) - INFO - Uploading, attempt #1.
14:32:53     INFO -  (blobuploader) - CRITICAL - Unexpected error in client: Traceback (most recent call last):
14:32:53     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 87, in upload_file
14:32:53     INFO -      ret = post_file(host, auth, filename, branch, hashalgo, blobhash)
14:32:53     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 151, in post_file
14:32:53     INFO -      verify=cert.where())
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/api.py", line 88, in post
14:32:53     INFO -      return request('post', url, data=data, **kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
14:32:53     INFO -      return session.request(method=method, url=url, **kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
14:32:53     INFO -      resp = self.send(prep, **send_kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
14:32:53     INFO -      r = adapter.send(request, **kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
14:32:53     INFO -      raise ConnectionError(e)
14:32:53     INFO -  ConnectionError: HTTPSConnectionPool(host='blobupload.elasticbeanstalk.com', port=443): Max retries exceeded with url: /blobs/sha512/210b959b0d45c8b110629e28aa39a93e22d26d43ecb583167e24ffe7f5ca00cec98cc43d605c03b01eb7700952781ff8020132c380ad522863202244f1a4997c (Caused by <class 'httplib.BadStatusLine'>: '')
14:32:53     INFO -  (blobuploader) - INFO - Done attempting.

As a side-note it seems only Ash has the traceback. All other branches just say "CRITICAL - Something went wrong on blobber!". I believe this is because the other branches are using an older version of blobuploader that doesn't contain https://github.com/MihaiTabara/blobuploader/commit/fd92482fe5485be89e238fc4fdb2c4688e37f763 .. but I could be wrong (which would mean there are two separate issues).
See https://bugzilla.mozilla.org/show_bug.cgi?id=956672#c6 for the root clause causing the server to return 500.

Probably we should make the server python 2.6 compatible to avoid EB hacks.

I think, we don't block bug 956672 anymore.
No longer blocks: 956672
Component: General Automation → Tools
QA Contact: catlee → hwine
Summary: Blobber fails to upload logcat | ConnectionError: HTTPSConnectionPool(host='blobupload.elasticbeanstalk.com', port=443) → Make blobber python 2.6 compatible
Whiteboard: [blobber]
It turns out that EB supports Python 2.7 by default now. \o/
Assignee: nobody → rail
Summary: Make blobber python 2.6 compatible → Migrate blobber server to python 2.7 Elastic Beanstalk
I created 2 new environments with newer EB AMIs: blobupload27 and blobupload27-1. Both work as expected. There are some changes waiting to be merged in https://github.com/catlee/blobber/pull/5.
All done here.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: Tools → General
You need to log in before you can comment on or make changes to this bug.