1015934 - Failure "TypeError: request() got an unexpected keyword argument 'config'" on amazon dnsless spot instances

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

Android 2.2 mozilla-central nightly on 2014-05-26 03:11:51 PDT for push e86a0d92d174

slave: bld-linux64-spot-1008

https://tbpl.mozilla.org/php/getParsedLog.php?id=40396019&tree=Mozilla-Central

TypeError: request() got an unexpected keyword argument 'config'

Comment 1

[Pete Moore [:pmoore][:pete]]

From a superficial inspection, it looks like a conflict between parameters passed here: 

  * https://hg.mozilla.org/build/tools/file/0fe254e5e200/lib/python/balrog/submitter/api.py#l115

and the allowed params here:

  * http://docs.python-requests.org/en/latest/api/#requests.Session.request

Notably, we are passing a parameter called "config", which is not expected by the function.

However, I don't understand why we didn't always get this problem - since this code seems to have gone live around two months ago...

However, I haven't dug very deep, because I know Ben will be online shortly to help out ;)

Pete

Comment 2

[[e-mail] (:bhearsum)]

We haven't upgraded the "requests" library in build/tools in ages (it's using 0.10.8: https://github.com/mozilla/build-tools/tree/master/lib/python/vendor/requests-0.10.8) - there must be a different one getting into PYTHONPATH somehow. Did any new system libraries get installed or some such?

Comment 3

[Carsten Book [:Tomcat]]

since this affects nightlys on android on serveral trees like beta/aurora i reset the severity

Comment 4

[[e-mail] (:bhearsum)]

Looks like this is only a problem with the new dnsless EC2 machines. They have a very new Requests installed in the system Python:
[[e-mail] ~]$ rpm -qa | grep requests
python-requests-1.1.0-4.el6.noarch


Whereas other machines don't:
[[e-mail] ~]$ rpm -qa | grep requests
[[e-mail] ~]$

Comment 5

[Rail Aliiev [:rail]]

We use site.addsitedir(...) to add paths to our "lib/python" directory which contains vendorlibs.pth with paths to the vendor packages we use. The site module adds the libraries we need but doesn't instert them prior to the system wide packages.

This is how mod_wsgi manages their sys.path (from https://code.google.com/p/modwsgi/wiki/VirtualEnvironments):

ALLDIRS = ['usr/local/pythonenv/PYLONS-1/lib/python2.5/site-packages']

import sys 
import site 

# Remember original sys.path.
prev_sys_path = list(sys.path) 

# Add each new site-packages directory.
for directory in ALLDIRS:
  site.addsitedir(directory)

# Reorder sys.path so new directories at the front.
new_sys_path = [] 
for item in list(sys.path): 
    if item not in prev_sys_path: 
        new_sys_path.append(item) 
        sys.path.remove(item) 
sys.path[:0] = new_sys_path

Comment 6

[Rail Aliiev [:rail]]

Attachment: path.diff

WCPGW?

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 8428736 [details] [diff] [review]
path.diff

Review of attachment 8428736 [details] [diff] [review]:
-----------------------------------------------------------------

Sucks, but necessary...can you make sure there's a bug on file to get using non-emebbed libs?

Thanks for fixing this.

Comment 8

[Justin Wood (:Callek)]

From phone. Can't we use the requests module not in system python?  I'm not really a fan of anything extra in system python directory. Always causes confusion.  (Sent from phone)

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 8428736 [details] [diff] [review]
path.diff

https://hg.mozilla.org/build/tools/rev/fce4f641249a