762399 - Run sync1.1 functional tests against a live server

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Ryan Kelly [:rfkelly]]

Attachment: server-storage patch to run tests against a live server

This is similar in spirit to Bug 731511, but targeting the old server-storage codebase.  It's a little less elegant because you need to provide a username and password the exists on the server, but should be useful enough for our purposes.

Run it like so:

    ./bin/python syncstorage/tests/functional/test_storage.py http://sync.server.com/

It'll prompt for username and password on the console.  You can also specify them as command-line arguments, use --help for the details.

Comment 1

[James Bonacci [:jbonacci]]

Very interesting. Will give this a try this week (hopefully)...

Comment 2

[Rob Miller [:rmiller]]

Comment on attachment 630886 [details] [diff] [review]
server-storage patch to run tests against a live server

Review of attachment 630886 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM!

Comment 3

[Ryan Kelly [:rfkelly]]

http://hg.mozilla.org/services/server-storage/rev/d35f6125fda4

Comment 4

[James Bonacci [:jbonacci]]

Basic results look really good, once I got the command line looking correct.
I was able to hit Stage with 3 failures and 1 error.
I was able to hit Prod but was denied access (more or less).

Command line:
./bin/python deps/server-storage/syncstorage/tests/functional/test_storage.py HOST
where HOST is the actual sync server being hit in Stage or Prod
Supply the encrypted user name for that account and its password

Alternatively:
./bin/python deps/server-storage/syncstorage/tests/functional/test_storage.py --username=<encrypted user name> --password=<test-based password> HOST

Leaving this Resolved/Fixed with :rfkelly looks at Stage and Prod errors/failures/issues...

Comment 5

[Ryan Kelly [:rfkelly]]

(The failures reported to me via private email to avoid disclosing passwords etc, I am including just the names and errors below.)

I believe these failures are a combination of undeployed bugfixes, and the tests expecting local computer time to match the server time.

> ======================================================================
> ERROR: test_handling_of_invalid_json (__main__.LiveTestStorage)
> ----------------------------------------------------------------------
> 
> AppError: Bad response: 503 Retry-Later (500) (not 400)

I believe this is caused by a lack of input validation.  This was added in Bug 691717 but has not yet been deployed.  Once Bug 761068 completes the failure should go away.

> ======================================================================
> FAIL: test_batch (__main__.LiveTestStorage)
> ----------------------------------------------------------------------
> 
> AssertionError: 105 != 100

This tests for new behaviour added in Bug 692372, which is not yet deployed to stage.  Once Bug 761068 completes the failure should go away.

> ======================================================================
> FAIL: test_delete_storage (__main__.LiveTestStorage)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "deps/server-storage/syncstorage/tests/functional/test_storage.py", line 583, in
> test_delete_storage
>    self.assertTrue(abs(now - float(res)) < 0.5)
> AssertionError

This is almost certainly caused by the increased latency of talking to a real server over the network.  Changing it to a simple "float(res) < now" causes the test to pass.

> ======================================================================
> FAIL: test_strict_newer_tabs (__main__.LiveTestStorage)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>  File "deps/server-storage/syncstorage/tests/functional/test_storage.py", line 874, in
> test_strict_newer_tabs
>     self.assertEquals(res, ['3', '4'])
> AssertionError: [u'1', u'3', u'2', u'4'] != ['3', '4']

Aha!  This is Bug 745062, which was fixed in sync2.0 but not in sync1.1.  I've backported the patch from that bug and will include it in the next release.


I also got some additional timing-related failures, and have filed Bug 762779 as a reminder to fix them.

Comment 6

[Ryan Kelly [:rfkelly]]

Attachment: patch to send a compatible Host header

Comment 7

[James Bonacci [:jbonacci]]

Here is the latest using Sync 1.1 code and Stage sync running the following:
python26-syncstorage-1.12-2.noarch



..E.............................
======================================================================
ERROR: test_batch (__main__.LiveTestStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "deps/server-storage/syncstorage/tests/functional/test_storage.py", line 694, in test_batch
    res = self.app.post_json(self.root + '/storage/col2', wbos)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 821, in post_json
    content_type=content_type)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 783, in _gen_request
    expect_errors=expect_errors)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 1062, in do_request
    self._check_status(status, res)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 1098, in _check_status
    res)
AppError: Bad response: 413 Request Entity Too Large (not 200 OK or 3xx redirect for https://stage-sync52.services.mozilla.com/1.0/4bih2appwpp5zqyw5tatghymjknwtq4s/storage/col2)
"client issue: request body too large"

----------------------------------------------------------------------
Ran 32 tests in 110.475s

FAILED (errors=1)

Comment 8

[Ryan Kelly [:rfkelly]]

(In reply to James Bonacci [:jbonacci] from comment #7)
> AppError: Bad response: 413 Request Entity Too Large (not 200 OK or 3xx
> redirect for
> https://stage-sync52.services.mozilla.com/1.0/
> 4bih2appwpp5zqyw5tatghymjknwtq4s/storage/col2)
> "client issue: request body too large"

I don't think this error is coming from the python code.  :atoll, can you please confirm whether this message is produced by Zeus and, if so, what the limit on request body size is?

Comment 9

[James Bonacci [:jbonacci]]

So, with Sync 1.1 code and a Prod account, I still see access errors:
All 32 tests fail with something similar to this:
AppError: Bad response: 401 Unauthorized (not 200 OK or 3xx redirect for https://BLAH/1.0/BLAH/storage)
401 Unauthorized

This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.

Comment 10

[James Bonacci [:jbonacci]]

OK, once I picked a working Prod account, I now see this:
..F..............E........F.....
======================================================================
ERROR: test_handling_of_invalid_json (__main__.LiveTestStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "deps/server-storage/syncstorage/tests/functional/test_storage.py", line 954, in test_handling_of_invalid_json
    status=400)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 857, in put_json
    content_type=content_type)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 783, in _gen_request
    expect_errors=expect_errors)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 1062, in do_request
    self._check_status(status, res)
  File "/opt/sync/server-full/lib/python2.6/site-packages/webtest/app.py", line 1101, in _check_status
    "Bad response: %s (not %s)", res_status, status)
AppError: Bad response: 503 Retry-Later (500) (not 400)

======================================================================
FAIL: test_batch (__main__.LiveTestStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "deps/server-storage/syncstorage/tests/functional/test_storage.py", line 688, in test_batch
    self.assertEquals(len(res['success']), 100)
AssertionError: 105 != 100

======================================================================
FAIL: test_strict_newer_tabs (__main__.LiveTestStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "deps/server-storage/syncstorage/tests/functional/test_storage.py", line 873, in test_strict_newer_tabs
    self.assertEquals(res, ['3', '4'])
AssertionError: [u'1', u'3', u'2', u'4'] != ['3', '4']

----------------------------------------------------------------------
Ran 32 tests in 115.891s

FAILED (failures=2, errors=1)

Comment 11

[Ryan Kelly [:rfkelly]]

Great, all three of these correspond to known server bugs/updates that will roll out as part of Bug 761068.  So, those failures are expected at this point.

Comment 12

[James Bonacci [:jbonacci]]

OK. Marking this as Verified/Fixed.

Comment 13

[:Atoll]

"client issue: request body too large" is indeed Zeus, i don't know the associated client bug #.