494635 - Sync engine test infrastructure needed

Status: RESOLVED DUPLICATE of bug 553401

(Firefox :: Sync, defect)

Description

[Anant Narayanan [:anant]]

Currently there is no way to properly test sync engines. Infrastructure that will allow us to verify the correctness of a sync engine would be of great help.

Comment 1

[Mike Connor [:mconnor]]

Ideally we'd use the staging cluster for this, since that'd also mean that whatever automated testing we have would break if the server broke in an incompatible way.

Should this be in the Server component?

Comment 2

[Ed Lee :Mardak]

If this is about running sync engine tests, we already use the js httpd local server for various tests. So we should be able to do that.. ?

Comment 3

[Mike Connor [:mconnor]]

Can we run the Weave server on the js httpd?  I wouldn't think that'd be practical...

Comment 4

[Ed Lee :Mardak]

I haven't looked at how it's implemented, but there are tests like test_records_wbo.js that do new Resource(..).get. But I don't see any post/puts..

Comment 5

[Dan Mills [:thunder]]

Oops, sorry for not chiming in here before.

I think we should do this with the js httpd.  We've tried to set up complete roundtrip tests before and failed, because you end up rat-holing on scripting the sync and trying to figure out how to simulate multiple clients.  It's not worth it.  Instead, let's just test the interaction of a single client & server; we can fully control the state of both before and after running a single sync.  We don't need to implement the full server API--for example:

Test 0:
* Server has nothing
* Client has nothing

We run a sync and check that keys have been uploaded to the server.  We only need to implement handlers for:

GET keys/public -> 404
GET keys/private -> 404
GET crypto/bookmarks -> 404
PUT keys/public
PUT keys/private
PUT crypto/bookmarks

(note: this would break if bug 479856 is fixed)

Test 1:
* Server has nothing
* Client has one bookmark

Like the above, plus we implement handlers for:

GET bookmarks?full=1 -> []
POST bookmarks

Test 2:
* Server has one bookmark
* Client has nothing

We implement one handler and make sure the bookmark ends up in the local db:

GET bookmarks?full=1 -> [...]

Open questions:

* can we abstract any of this so we can reuse it for multiple engines?
* what will we do once we have random key names? (maybe we can hardcode them for the test?)
* should we consider disabling encryption (not currently supported) for tests?  it would make debugging easier in some situations

Comment 6

[Mike Connor [:mconnor]]

(In reply to comment #5)
> Oops, sorry for not chiming in here before.
> 
> I think we should do this with the js httpd.  We've tried to set up complete
> roundtrip tests before and failed, because you end up rat-holing on scripting
> the sync and trying to figure out how to simulate multiple clients.  It's not
> worth it.  Instead, let's just test the interaction of a single client &
> server; we can fully control the state of both before and after running a
> single sync.  We don't need to implement the full server API--for example:

I think the further we get from real-world behaviours, the less useful the testing ends up being.  We shouldn't need to rathole on the complex cases, but it shouldn't be hard to do those three tests using test accounts.  We might need to have some set of base server-side files that we can copy into place at the start of a test run, but that shouldn't be very hard to invoke on the test server...

> Open questions:
> 
> * can we abstract any of this so we can reuse it for multiple engines?

In what sense?  The login/logout pieces should be the same, and ideally we can reuse that stuff, but the engine-specific stuff probably isn't going to be reusable if we're writing tests that are actually checking properties of specific items that are being synced back and forth.

> * what will we do once we have random key names? (maybe we can hardcode them
> for the test?)

Why would the key names matter?  Wouldn't we just call a high-level sync method and let the rest just work?

> * should we consider disabling encryption (not currently supported) for tests? 
> it would make debugging easier in some situations

Isn't that just introducing a different codepath than what users actually would use?

Comment 7

[Mike Connor [:mconnor]]

moving beta blockers to block 1.0 beta

Comment 9

[Richard Newman [:rnewman]]

This is a dupe; removing flags.