765656 - Use mockmyid.com for running functional tests against a tokenserver

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

Attachment: patch to use mockmyid.com for functional tests

Currently server-syncstorage runs functional tests against a live tokenserver by generating special "loadtest.local" assertions with a known private key.  This means it will only work if the tokenserver is configured to support such assertions.

To allow this to work with any random tokenserver setup, we should generate assertions signed by mymockid.com.  This is a real live browserid-supporting domain whose private key we happen to know.

Comment 1

[James Bonacci [:jbonacci]]

I thought we were already doing this for tokenserver load testing, so this seems like a good addition to that.

Comment 2

[Ryan Kelly [:rfkelly]]

Fixing strange extra like word in the bug summary.  Something is up with my typing and spelling lately...

Comment 3

[Toby Elliott [:telliott]]

Comment on attachment 633956 [details] [diff] [review]
patch to use mockmyid.com for functional tests

This is fine, but I wonder if we shouldn't consider hosting one ourselves, especially given what we likely want for browserid, as well as protecting us from mockmyid from going rogue or something. But that's orthogonal to this chunk of code.

Comment 4

[Richard Newman [:rnewman]]

So... how can we get this set up to have a Sync server which uses this? :D

Comment 5

[Ryan Kelly [:rfkelly]]

Committed in https://github.com/mozilla-services/server-syncstorage/commit/05a35f0310474f3f586e6fb108378d0541638a3a

:rnewman, once AITC deployment is out the door I will work on getting this running on the sync dev machines, in parallel with sync1.1 server.  In the meantime it would be great if you can try out the experimental run-your-own sync2 server and let me know how it works (or doesn't, case the case may be):

    https://github.com/mozilla-services/server-full2

Comment 6

[James Bonacci [:jbonacci]]

Reminder to me that this needs to be tested against aitc 1.2 and sync 2.0.

Comment 7

[James Bonacci [:jbonacci]]

Scratch that. The following bug covers aitc 1.2: 751485

Comment 8

[James Bonacci [:jbonacci]]

Have not tried this on sync 2.0 yet, but I am trying out the following repo:
https://github.com/mozilla-services/server-full2
on CentOS 6.3 (latest)

My system already has most of the pre-reqs including M2Crypto and Swig, but I am seeing the following during "make build":
...etc...
Searching for M2Crypto
Reading http://pypi.python.org/simple/M2Crypto/
Reading http://wiki.osafoundation.org/bin/view/Projects/MeTooCrypto
Reading http://www.post1.com/home/ngps/m2
Reading http://sandbox.rulemaker.net/ngps/m2/
Reading http://chandlerproject.org/Projects/MeTooCrypto
Best match: M2Crypto 0.21.1
Downloading http://pypi.python.org/packages/source/M/M2Crypto/M2Crypto-0.21.1.tar.gz#md5=f93d8462ff7646397a9f77a2fe602d17
Processing M2Crypto-0.21.1.tar.gz
Writing /tmp/easy_install-EiahUI/M2Crypto-0.21.1/setup.cfg
Running M2Crypto-0.21.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-EiahUI/M2Crypto-0.21.1/egg-dist-tmp-6J2mWg

SyntaxError: ('invalid syntax', ('build/bdist.linux-x86_64/egg/requests/packages/chardet2/test.py', 8, 27, "    print(f.ljust(60), end=' ')\n"))

SyntaxError: ('invalid syntax', ('/opt/sync-exp/server-full2/lib/python2.6/site-packages/requests-0.13.2-py2.6.egg/requests/packages/chardet2/test.py', 8, 27, "    print(f.ljust(60), end=' ')\n"))

/usr/include/openssl/opensslconf.h:31: Error: CPP #error ""This openssl-devel package does not work your architecture?"". Use the -cpperraswarn option to continue swig processing.
error: Setup script exited with error: command 'swig' failed with exit status 1

                                                                                               [fail]                                                                                               [fail]                                                                                               [fail]
make: *** [build] Error 1

$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 14:18:47) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import M2Crypto
>>> print M2Crypto.__file__
/usr/lib64/python2.6/site-packages/M2Crypto/__init__.pyc
>>> 


Going to try this on qa3 (rhel6) and see if I can get any further...

Comment 9

[James Bonacci [:jbonacci]]

OK, once I got past the libmemcached/libmemcached-devel/pylibmc issues, I got the same error on rhel 6.1.

Comment 10

[James Bonacci [:jbonacci]]

Results for my CentOS 6.3 VM: I got a 403 on the testing...
$ make test
bin/nosetests -s deps/server-syncstorage/syncstorage/tests
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 36.026s
OK

$ make serve
See http://jbonacci.pastebin.mozilla.org/1706626

Comment 11

[James Bonacci [:jbonacci]]

On r6.build.mtv1.svc, I get this for $make serve
http://jbonacci.pastebin.mozilla.org/1706631

My guess is somebody else is on the same VM doing stuff...

Comment 12

[James Bonacci [:jbonacci]]

qa3 (rhel6):
$ make test
bin/nosetests -s deps/server-syncstorage/syncstorage/tests
...................................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS...SSSSSSSSS..............
----------------------------------------------------------------------
Ran 96 tests in 70.473s
OK (SKIP=44)

Then the following for $ make serve, etc...
http://jbonacci.pastebin.mozilla.org/1706634

Comment 13

[Ryan Kelly [:rfkelly]]

From the pastebin:
> requests.exceptions.HTTPError: 403 Client Error 

This is due to tokenserver ToS changes, should be fixed by Bug 774175.  Please do a rebuild to get the latest changes and try again.

Comment 14

[James Bonacci [:jbonacci]]

Not seeing the 403 Client Error any more.

But, with the latest build, I see two types of errors.
With memcached off, I am seeing this on qa3, rh6, centos:
http://jbonacci.pastebin.mozilla.org/1715232


With memcached started/running on qa3 and centos (I have not permissions to control that on rh6), I see this instead:
http://jbonacci.pastebin.mozilla.org/1715249

I have opened a separate bug for the memcached-related errors and skips: bug 777625.

Comment 15

[Ryan Kelly [:rfkelly]]

:jbonacci the dependant bugs have now been fixed, please try running these tests again.

Comment 16

[James Bonacci [:jbonacci]]

Yep. Verified the depending bugs.
Verying this as fixed with the latest code.