Closed Bug 789972 Opened 12 years ago Closed 12 years ago

strange error on windows with dzclient

Categories

(Testing :: Talos, defect)

Product:
Testing
Component:
Talos
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: k0scist, Unassigned)

References

Details

NOISE: datazilla: https//datazilla.mozilla.org/test; oauth=True
Traceback (most recent call last):
  File "run_tests.py", line 298, in ?
    main()
  File "run_tests.py", line 295, in main
    run_tests(parser)
  File "run_tests.py", line 271, in run_tests
    talos_results.output(results_urls, **results_options)
  File "c:\talos-slave\talos-data\talos\results.py", line 78, in
  output
    _output.output(results, url)
  File "c:\talos-slave\talos-data\talos\output.py", line 391, in
  output
    self.post(results, results_server, results_path, results_scheme)
  File "c:\talos-slave\talos-data\talos\output.py", line 472, in post
    responses = req.submit()
  File "c:\talos-slave\talos-data\talos\dzclient.py", line 194, in
  submit
    responses.append(self.send(dataset))
  File "c:\talos-slave\talos-data\talos\dzclient.py", line 225, in
  send
    req = oauth.Request(method="POST", url=uri, parameters=params)
  File "c:\talos-slave\talos-data\talos\oauth2\__init__.py", line 350,
  in __init__
    v = to_unicode_optional_iterator(v)
  File "c:\talos-slave\talos-data\talos\oauth2\__init__.py", line 135,
  in to_unicode_optional_iterator
    assert 'is not iterable' in str(e)
Blocks: 721097
It seems an assertion somehow makes its way to the output. Anything more you can throw out in addition to this stack trace?
So I have tried to reproduce this on windows with:

talos -n -d -a ts --cycles 1 --develop -e "c:\Program Files (x86)\Mozilla Firefox\firefox.exe" --datazilla-url https://datazilla.mozilla.org/test --authfile passwords.py

Of course...I can't reproduce and it works fine o_O

So I'm not sure what I can do here.

:bhearsum, can you provide the command line (or what not) that triggers this error?  Are you using talos trunk + datazilla_client trunk?

I can add debug output to datazilla_client to print uri and params if that helps.
This was using:
http://hg.mozilla.org/mozilla-central/raw-file/12dad118c02f/testing/talos/talos.json

The PerfConfigurator command line was:
'python' 'PerfConfigurator.py' '-v' '-e' '../firefox/firefox' '-t' 'talos-r3-w7-002' '--branchName' 'MozillaTest-Non-PGO' '--resultsServer' 'graphs.allizom.org' '--resultsLink' '/server/collect.cgi' '--datazilla-url' 'https://datazilla.mozilla.org/test' '--activeTests' 'tdhtmlr' '--noChrome' '--mozAfterPaint' '--filter' 'ignore_first:5' '--filter' 'median' '--symbolsPath' '../symbols'
(In reply to Ben Hearsum [:bhearsum] from comment #3)
> This was using:
> http://hg.mozilla.org/mozilla-central/raw-file/12dad118c02f/testing/talos/
> talos.json
> 
> The PerfConfigurator command line was:
> 'python' 'PerfConfigurator.py' '-v' '-e' '../firefox/firefox' '-t'
> 'talos-r3-w7-002' '--branchName' 'MozillaTest-Non-PGO' '--resultsServer'
> 'graphs.allizom.org' '--resultsLink' '/server/collect.cgi' '--datazilla-url'
> 'https://datazilla.mozilla.org/test' '--activeTests' 'tdhtmlr' '--noChrome'
> '--mozAfterPaint' '--filter' 'ignore_first:5' '--filter' 'median'
> '--symbolsPath' '../symbols'

I've tried:

talos -n -d -a tdhtmlr --cycles 1 --develop -e "c:\Program Files (x86)\Mozilla Firefox\firefox.exe" --datazilla-url https://datazilla.mozilla.org/test --authfile passwords.py -t talos-r3-w7-002 --branchName MozillaTest-Non-PGO --noChrome --mozAfterPaint --filter ignore_first:5 --filter median

This should roughly be equivalent.  Again, no errors.  This is still on tip talos, tip datazilla_client.  I will try to reproduce with the version referenced in comment 3 . Other than adding more error output to datazilla_client, which should probably be done, I'm not really sure what else I can do from my end.
(In reply to Jeff Hammel [:jhammel] from comment #4)
> (In reply to Ben Hearsum [:bhearsum] from comment #3)
> > This was using:
> > http://hg.mozilla.org/mozilla-central/raw-file/12dad118c02f/testing/talos/
> > talos.json
> > 
> > The PerfConfigurator command line was:
> > 'python' 'PerfConfigurator.py' '-v' '-e' '../firefox/firefox' '-t'
> > 'talos-r3-w7-002' '--branchName' 'MozillaTest-Non-PGO' '--resultsServer'
> > 'graphs.allizom.org' '--resultsLink' '/server/collect.cgi' '--datazilla-url'
> > 'https://datazilla.mozilla.org/test' '--activeTests' 'tdhtmlr' '--noChrome'
> > '--mozAfterPaint' '--filter' 'ignore_first:5' '--filter' 'median'
> > '--symbolsPath' '../symbols'
> 
> I've tried:
> 
> talos -n -d -a tdhtmlr --cycles 1 --develop -e "c:\Program Files
> (x86)\Mozilla Firefox\firefox.exe" --datazilla-url
> https://datazilla.mozilla.org/test --authfile passwords.py -t
> talos-r3-w7-002 --branchName MozillaTest-Non-PGO --noChrome --mozAfterPaint
> --filter ignore_first:5 --filter median
> 
> This should roughly be equivalent.  Again, no errors.  This is still on tip
> talos, tip datazilla_client.  I will try to reproduce with the version
> referenced in comment 3 . Other than adding more error output to
> datazilla_client, which should probably be done, I'm not really sure what
> else I can do from my end.

I'm happy to do what I can to help. If there's any patched code to try, or you need machine access, or different commands I could run - please let me know.
(In reply to Jeff Hammel [:jhammel] from comment #2)
<snip/>
> I can add debug output to datazilla_client to print uri and params if that
> helps.

https://github.com/mozilla/datazilla_client/issues/26
(In reply to Jeff Hammel [:jhammel] from comment #7)
> pull request at https://github.com/mozilla/datazilla_client/pull/27

merged; datazilla client 1.1 is on pypi: http://pypi.python.org/pypi/datazilla/1.1
up-to-the-minute talos with dependencies rolled into http://people.mozilla.com/~jhammel/talos.aba49a189add.zip
NOISE: Outputting datazilla results to https://datazilla.mozilla.org/test/
NOISE: datazilla: https//datazilla.mozilla.org/test; oauth=True
uri: https://datazilla.mozilla.org/test/api/load_test
params: {'oauth_nonce': '93370902', 'oauth_timestamp': 1347977107, 'oauth_consumer_key': '84c76536-5bea-4858-990b-434ad3bbadc1', 'oauth_version': '1.0', 'oauth_token': '', 'user': 'test', 'data': '%7B%22test_machine%22%3A%20%7B%22platform%22%3A%20%22x86%22%2C%20%22osversion%22%3A%20%226.1.7600%22%2C%20%22os%22%3A%20%22win%22%2C%20%22name%22%3A%20%22talos-r3-w7-001%22%7D%2C%20%22testrun%22%3A%20%7B%22date%22%3A%201347976110%2C%20%22suite%22%3A%20%22Talos%20tresize%22%2C%20%22options%22%3A%20%7B%22responsiveness%22%3A%20false%2C%20%22tpmozafterpaint%22%3A%20false%2C%20%22tpchrome%22%3A%20true%2C%20%22tppagecycles%22%3A%201%2C%20%22tpcycles%22%3A%2010%2C%20%22tprender%22%3A%20false%2C%20%22shutdown%22%3A%20false%2C%20%22extensions%22%3A%20%5B%7B%22name%22%3A%20%22pageloader%40mozilla.org%22%7D%5D%2C%20%22rss%22%3A%20false%7D%7D%2C%20%22results%22%3A%20%7B%22tresize-test.html<a bunch of stuff that won't fit here>.0%5D%7D%2C%20%22test_build%22%3A%20%7B%22version%22%3A%20%2218.0a1%22%2C%20%22revision%22%3A%20%2212dad118c02f%22%2C%20%22id%22%3A%20%2220120910070417%22%2C%20%22branch%22%3A%20%22MozillaTest-Non-PGO%22%2C%20%22name%22%3A%20%22Firefox%22%7D%7D'}
Traceback (most recent call last):
  File "run_tests.py", line 298, in ?
    main()
  File "run_tests.py", line 295, in main
    run_tests(parser)
  File "run_tests.py", line 271, in run_tests
    talos_results.output(results_urls, **results_options)
  File "c:\talos-slave\talos-data\talos\results.py", line 78, in output
    _output.output(results, url)
  File "c:\talos-slave\talos-data\talos\output.py", line 392, in output
    self.post(results, results_server, results_path, results_scheme)
  File "c:\talos-slave\talos-data\talos\output.py", line 472, in post
    responses = req.submit()
  File "c:\talos-slave\talos-data\talos\dzclient.py", line 194, in submit
    responses.append(self.send(dataset))
  File "c:\talos-slave\talos-data\talos\dzclient.py", line 226, in send
    req = oauth.Request(method="POST", url=uri, parameters=params)
  File "c:\talos-slave\talos-data\talos\oauth2\__init__.py", line 350, in __init__
    v = to_unicode_optional_iterator(v)
  File "c:\talos-slave\talos-data\talos\oauth2\__init__.py", line 135, in to_unicode_optional_iterator
    assert 'is not iterable' in str(e)
AssertionError
Depends on: 792092
Encountered https://bugzilla.mozilla.org/show_bug.cgi?id=792227 in the process of chasing this down.  Command line:

C:\Users\cltbld\test\talos>python run_tests.py -n -d -a tdhtmlr:tresize -e ../fi
refox/firefox.exe --datazilla-url https://datazilla.mozilla.org/test --authfile
../BuildSlaves.py
So, I've found the actual error which seems to be a python 2.4-ism:

oauth2.__init__ contains the following code:

def to_unicode_optional_iterator(x):
    """
    Raise TypeError if x is a str containing non-utf8 bytes or if x is
    an iterable which contains such a str.
    """
    if isinstance(x, basestring):
        return to_unicode(x)

    import pdb; pdb.set_trace()
    try:
        l = list(x)
    except TypeError, e:
        assert 'is not iterable' in str(e)
        return x
    else:
        return [ to_unicode(e) for e in l ]

Note that instead of looking at e.g. __iter__ which would make friggin sense, instead we cast to a list and raise an exception. "Nice".  This is a great example of why not to do that.  On my linux machine with python 2.7, doing

>>> list(12345)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable

has the magical string they are looking for.  However, on the build slave with python 2.4.4 I get 

>>> list(12345)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: iteration over non-sequence
>>>

Oops!  I'm not sure what to do about this yet.  But needless to say, I am not impressed.
Time to apply a monkeypatch? ;)
So it looks like we can encode the dictionary ourselves and avoid the bug:

https://github.com/simplegeo/python-oauth2/blob/master/oauth2/__init__.py#L328

I'll also make a pull request for the upstream case
(In reply to Jeff Hammel [:jhammel] from comment #14)
> So it looks like we can encode the dictionary ourselves and avoid the bug:
> 
> https://github.com/simplegeo/python-oauth2/blob/master/oauth2/__init__.
> py#L328
> 
> I'll also make a pull request for the upstream case

pull request submitted: https://github.com/simplegeo/python-oauth2/pull/120
Fixing this code (locally).  Now I get:

Traceback (most recent call last):
  File "run_tests.py", line 298, in ?
    main()
  File "run_tests.py", line 295, in main
    run_tests(parser)
  File "run_tests.py", line 271, in run_tests
    talos_results.output(results_urls, **results_options)
  File "C:\Users\cltbld\test\talos\results.py", line 78, in output
    _output.output(results, url)
  File "C:\Users\cltbld\test\talos\output.py", line 392, in output
    self.post(results, results_server, results_path, results_scheme)
  File "C:\Users\cltbld\test\talos\output.py", line 472, in post
    responses = req.submit()
  File "C:\Users\cltbld\test\talos\dzclient.py", line 194, in submit
    responses.append(self.send(dataset))
  File "C:\Users\cltbld\test\talos\dzclient.py", line 237, in send
    req.sign_request(signature_method, consumer, token)
  File "C:\Users\cltbld\test\talos\oauth2\__init__.py", line 491, in sign_reques
t
    digested = sha(self.body).digest()
TypeError: 'module' object is not callable

This is because https://github.com/simplegeo/python-oauth2/blob/master/oauth2/__init__.py#L45 :

try:
    from hashlib import sha1
    sha = sha1
except ImportError:
    # hashlib was added in Python 2.5
    import sha

Should be 'from sha import sha' . "Oops!"  Will ticket this as well
Working around this, now I get...

NOISE: datazilla: https//datazilla.mozilla.org/test; oauth=True
Traceback (most recent call last):
  File "run_tests.py", line 298, in ?
    main()
  File "run_tests.py", line 295, in main
    run_tests(parser)
  File "run_tests.py", line 271, in run_tests
    talos_results.output(results_urls, **results_options)
  File "C:\Users\cltbld\test\talos\results.py", line 78, in output
    _output.output(results, url)
  File "C:\Users\cltbld\test\talos\output.py", line 392, in output
    self.post(results, results_server, results_path, results_scheme)
  File "C:\Users\cltbld\test\talos\output.py", line 472, in post
    responses = req.submit()
  File "C:\Users\cltbld\test\talos\dzclient.py", line 194, in submit
    responses.append(self.send(dataset))
  File "C:\Users\cltbld\test\talos\dzclient.py", line 237, in send
    req.sign_request(signature_method, consumer, token)
  File "C:\Users\cltbld\test\talos\oauth2\__init__.py", line 499, in sign_reque
t
    self['oauth_signature'] = signature_method.sign(self, consumer, token)
  File "C:\Users\cltbld\test\talos\oauth2\__init__.py", line 837, in sign
    hashed = hmac.new(key, raw, sha)
  File "c:\Python24\lib\hmac.py", line 107, in new
    return HMAC(key, msg, digestmod)
  File "c:\Python24\lib\hmac.py", line 42, in __init__
    self.outer = digestmod.new()
AttributeError: 'builtin_function_or_method' object has no attribute 'new'
So this one is fairly bizarre ;)

* oauth2 calls hmac.new(key, raw, sha)
  https://github.com/simplegeo/python-oauth2/blob/master/oauth2/__init__.py#L840

* in the python 2.4 API, hmac.new expects the third, argument, digestmod, to have a .new() callable: http://svn.python.org/view/python/tags/r244/Lib/hmac.py?view=markup .  This is probably why the sha module was imported, not the function, leading to the problem from comment 16

* in the python 2.5 API, if the digestmod pass in is a callable, we use that instead of looking for a .new attribute on it: http://svn.python.org/view/python/branches/release25-maint/Lib/hmac.py?view=markup

* I fixed comment 16 with https://github.com/k0s/python-oauth2/commit/57003712cd2955f812d7ab46d3531376c26cc9ed which lead to comment 17 and this problem

The fix is up at https://github.com/k0s/python-oauth2/commit/dd34c70b0cc8249cb5f7837540ad07001ae858e8
I've submitted pull requests for these issues.  Sadly, they're mostly not monkey-patchable.  On the bright side, they're all python 2.4 only.  Since we hope to EOL python 2.4 soon anyway, and since it should only apply via the talos zip, generated by http://hg.mozilla.org/build/talos/file/38609147b9db/create_talos_zip.py , my current suggestion (unless I find more broken stuff) is just to point the oauth files at my repo vs simplegeo's repo: http://hg.mozilla.org/build/talos/file/38609147b9db/create_talos_zip.py#l80

I'm not a fan of such a loose vendor-branch strategy, but since all of the nonsense wrt both python 2.4 and create_talos_zip.py should (and had better!) go away soon, in this case it seems the least evil alternative unless simplegeo lands my pull request
So the good news is....with a locally very hacked oauth with the above issues, I have been able to post test data from the build slave.

The bad news is....all of these errors come from oauth2 incompatabilities with python 2.4.  Since this is 3rd party code....this is somewhat annoying :(

The not so bad news is....since talos on windows/py24 only runs out of talos.zip, we can put in a forked branch of the oauth2 files as suggested in comment 19 . I'm going to use my fork at https://github.com/k0s/python-oauth2 to update create_talos_zip.py . If that works for the build slave, I will make that official.  Then, once we get off of python 2.4, we can switch back...and then once talos.zip goes away, even better :)
Depends on: 792989
there is a tarball that seems to work at http://people.mozilla.com/~jhammel/talos.4e83b813a258.zip
(In reply to Jeff Hammel [:jhammel] from comment #21)
> there is a tarball that seems to work at
> http://people.mozilla.com/~jhammel/talos.4e83b813a258.zip

No problems when running through Buildbot with this revision, hooray!
I'm going to close this then.  The work here seems done
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.