This patch refactors the live-functional-tests runner so that it can use tokens obtained from a live tokenserver. Currently the only option is to synthesize tokens using a known shared secret, which makes it difficult to run the tests against stage/prod where the secret files are locked down. The default usage remains the same - when invoked with just a URL it will synthesize tokens using the default config and run the tests that way: python aitc/tests/functional/test_aitc.py http://localhost:5000/ If you pass the --use-token-server option, then it will treat the given URL as a tokenserver rather than an endpoint, like this: python aitc/tests/functional/test_aitc.py --use-token-server https://stage-token.services.mozilla.com/1.0/aitc/1.0 Try python aitc/tests/functional/test_aitc.py --help for all the available options. I have used this to successfully run the AITC functional tests against the latest stage deployment. Thoughts?
Assignee: nobody → rfkelly
Comment on attachment 620610 [details] [diff] [review] server-syncstorage patch to run tests using live tokenserver This seems like a good approach to the problem. I'll note that there's the unlikely possibility that the token could expire during the tests if you had a really long set of tests or a really short-lived token. I doubt this is something that needs worrying about.
Attachment #620610 - Flags: feedback?(telliott) → feedback+
Comment on attachment 620610 [details] [diff] [review] server-syncstorage patch to run tests using live tokenserver You will get a +1 from me on this. QA has been doing this manually - pointing qa env to Dev env - by grabbing the secrets file off of Dev and making some changes to the config. I also like that it is command-line driven, but are there other steps that would have to be taken get this to work? That is, any specific changes on the source side (like the QA env or a local install). Also, can this idea be generalized to get an aitc loadtest working against a live tokenserver?
(In reply to James Bonacci [:jbonacci] from comment #3) > I also like that it is command-line driven, but are there other steps that > would have to be taken get this to work? That is, any specific changes on > the source side (like the QA env or a local install). You need to run it from an environment produced by `make build`. There's one complication: this patch has a dependency on PyBrowserID, and hence on M2Crypto. So you'll need to do the usual symlink dance to get a working M2Crypto in your virtualenv. So it'd be something like: make build ln -s /usr/lib64/python2.6/site-packages/M2Crypto* ./lib/python2.6/site-packages/ ./bin/pip install PyBrowserID ./bin/python aitc/tests/functional/test_aitc.py --use-token-server <server-url> The patch currently imports PyBrowserID at runtime only if needed for the test, specifically to *avoid* making M2Crypto a hard dependency for syncstorage/AITC. /me seriously considers replacing M2Crypto with PyCrypto for easy installability...
I don't suppose it would be too much to ask for a test mode that didn't hook up to BrowserID? I would really like the mock JS server to not have any external dependencies. Furthermore, I'd love to run a subset of the functional tests against it which didn't rely on external services. Feel free to call me crazy.
The default mode of running the tests doesn't hook up to BrowserID, it creates its own tokens. Using full-blown BrowserID assertions is only enabled with the --use-token-server argument. So you should still be able to use this to test the Mock JS Server as before: python aitc/tests/function/test_aitc.py https://localhost:8080/
(In reply to Ryan Kelly [:rfkelly] from comment #6) > The default mode of running the tests doesn't hook up to BrowserID, it > creates its own tokens. Using full-blown BrowserID assertions is only > enabled with the --use-token-server argument. > > So you should still be able to use this to test the Mock JS Server as before: > > python aitc/tests/function/test_aitc.py https://localhost:8080/ \o/
:rfkelly, in ref to Comment 6 I only tried that technique with Sync 2.0 code. Is there an aitc equivalent to syncstorage/tests/tests.ini that needs to be configured? Or can you start the server straight away on one terminal and run the line above on another? Example from Sync 2.0: ./bin/paster serve syncstorage/tests/tests.ini ./bin/python syncstorage/tests/functional/test_storage.py http://localhost:5050/
(In reply to James Bonacci [:jbonacci] from comment #8) > Example from Sync 2.0: > ./bin/paster serve syncstorage/tests/tests.ini > ./bin/python syncstorage/tests/functional/test_storage.py > http://localhost:5050/ The equivalent should also work for AITC: ./bin/paster serve aitc/tests/tests.ini ./bin/python aitc/tests/functional/test_aitc.py http://localhost:5000/
OK, updating patches with some small code cleanups but the underlying idea remains the same.
Attachment #620950 - Flags: review?(telliott) → review+
Attachment #620951 - Flags: review?(telliott) → review+
I'd like to remove the need for M2Crypto before landing this, see https://github.com/mozilla/PyBrowserID/pull/4 to track that effort.
:telliott and :rfkelly - let me know when this gets to a "QA testable" stage (whether or not it's marked Resolved). I want to schedule this in with some other AITC testing.
Committed in: https://github.com/mozilla-services/server-syncstorage/commit/fff34a7afc730898e1c49821c32ca05ab4733d4b https://github.com/mozilla-services/server-aitc/commit/1837378688dfac886d64549edc0e96c5f382e381 To get it running you'll need to do the M2Crypto install dance. The following steps work for me on r6-build: make build ln -sf `python -c 'import M2Crypto; print M2Crypto.__file__' | xargs dirname`* ./lib/*/site-packages ./bin/pip install PyBrowserID ./bin/python aitc/tests/functional/test_aitc.py --use-token-server https://stage-token.services.mozilla.com/1.0/aitc/1.0
Once the latest RPMs are pushed to Stage, I can try this from qa1 and from one of the clientX boxes.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Verified this is working as expected in Stage. Given the instructions in Comment 14 I see one error: ERROR: test_that_uploading_invalid_json_gives_a_400_response (syncstorage.tests.functional.support.LiveTestCases) ---------------------------------------------------------------------- Traceback (most recent call last): File "aitc/tests/functional/test_aitc.py", line 389, in test_that_uploading_invalid_json_gives_a_400_response self.app.put_json(self.root + "/apps/" + id, data, status=400) File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 857, in put_json content_type=content_type) File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 783, in _gen_request expect_errors=expect_errors) File "/opt/aitc1/server-aitc/deps/https:/github.com/mozilla-services/server-syncstorage/syncstorage/tests/functional/support.py", line 39, in new_do_request return orig_do_request(req, *args, **kwds) File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1062, in do_request self._check_status(status, res) File "/opt/aitc1/server-aitc/lib/python2.6/site-packages/webtest/app.py", line 1101, in _check_status "Bad response: %s (not %s)", res_status, status) AppError: Bad response: 201 Created (not 400) This is expected assuming that "ignore_unknown_fields being" is enabled on Stage. Per :rfkelly, it's accepting what would be an invalid request if it were to hit prod
Status: RESOLVED → VERIFIED
So this now works when pointing to the latest AITC code in Production: ./bin/python aitc/tests/functional/test_aitc.py --use-token-server https://token.services.mozilla.com/1.0/aitc/1.0 No errors in Production.
You need to log in before you can comment on or make changes to this bug.