958056 - Migrate blobber server to python 2.7 Elastic Beanstalk

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

Across all branches blobber is hitting a critical error. There is a traceback on Ash:
https://tbpl.mozilla.org/php/getParsedLog.php?id=32046578&tree=Ash&full=1#error0

14:32:41     INFO -  (blobuploader) - INFO - Uploading, attempt #1.
14:32:53     INFO -  (blobuploader) - CRITICAL - Unexpected error in client: Traceback (most recent call last):
14:32:53     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 87, in upload_file
14:32:53     INFO -      ret = post_file(host, auth, filename, branch, hashalgo, blobhash)
14:32:53     INFO -    File "/builds/slave/test/build/venv/bin/blobberc.py", line 151, in post_file
14:32:53     INFO -      verify=cert.where())
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/api.py", line 88, in post
14:32:53     INFO -      return request('post', url, data=data, **kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
14:32:53     INFO -      return session.request(method=method, url=url, **kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
14:32:53     INFO -      resp = self.send(prep, **send_kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
14:32:53     INFO -      r = adapter.send(request, **kwargs)
14:32:53     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
14:32:53     INFO -      raise ConnectionError(e)
14:32:53     INFO -  ConnectionError: HTTPSConnectionPool(host='blobupload.elasticbeanstalk.com', port=443): Max retries exceeded with url: /blobs/sha512/210b959b0d45c8b110629e28aa39a93e22d26d43ecb583167e24ffe7f5ca00cec98cc43d605c03b01eb7700952781ff8020132c380ad522863202244f1a4997c (Caused by <class 'httplib.BadStatusLine'>: '')
14:32:53     INFO -  (blobuploader) - INFO - Done attempting.

As a side-note it seems only Ash has the traceback. All other branches just say "CRITICAL - Something went wrong on blobber!". I believe this is because the other branches are using an older version of blobuploader that doesn't contain https://github.com/MihaiTabara/blobuploader/commit/fd92482fe5485be89e238fc4fdb2c4688e37f763 .. but I could be wrong (which would mean there are two separate issues).

Comment 1

[Rail Aliiev [:rail]]

See https://bugzilla.mozilla.org/show_bug.cgi?id=956672#c6 for the root clause causing the server to return 500.

Probably we should make the server python 2.6 compatible to avoid EB hacks.

I think, we don't block bug 956672 anymore.

Comment 2

[Rail Aliiev [:rail]]

It turns out that EB supports Python 2.7 by default now. \o/

Comment 3

[Rail Aliiev [:rail]]

I created 2 new environments with newer EB AMIs: blobupload27 and blobupload27-1. Both work as expected. There are some changes waiting to be merged in https://github.com/catlee/blobber/pull/5.

Comment 4

[Rail Aliiev [:rail]]

All done here.