1219456 - Stand up TPS in services infrastructure

Status: RESOLVED INCOMPLETE

(Cloud Services :: QA: Test Automation, defect)

Description

[Stuart Philp :sphilp]

https://developer.mozilla.org/en-US/docs/TPS

Looking at moving this into the services qa infrastructure so that :markh can take ownership of build and test, and use it for sync going forward.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I replied to the email which has been sent out by last week. I hope it gives you all necessary bits to get started. If not you can always ask me further questions.

Comment 2

[Karl Thiessen [:kthiessen, he/him]]

Attachment: Log showing failure of test when supplies with a valid FxA account

This is different from failures associated with bad username/password.

Comment 3

[Mark Hammond [:markh] [:mhammond]]

(In reply to Karl Thiessen [:kthiessen] from comment #2)
> This is different from failures associated with bad username/password.

That's a good result :) It looks like it reflects a relatively successful environment but an early test failure. I'll find (or open) the client-side bug to fix that tomorrow, but reproducing that failure in a CI environment would mean we are almost there.

Comment 4

[Chris Karlof [:ckarlof]]

Stuart, can we close this bug or is there further work to do here?

Comment 5

[Stuart Philp :sphilp]

I wouldn't mind keeping this open until we have all the pieces complete. although technically TPS runs in our jenkins, it's not 100% in CI building on changes and such and there's also reporting into treeherder (though that part can/should be a separate bug)

Comment 6

[Karl Thiessen [:kthiessen, he/him]]

Attachment: Test run showing Fxa/Hawk errors

This shows the output of a TPS run from the QA Jenkins box.

Comment 7

[Stuart Philp :sphilp]

Woo! Need to figure out the best way to get this to build on commit so markh can see progress

Comment 8

[Karl Thiessen [:kthiessen, he/him]]

Good news and bad news, as always.

Good news: We've gotten TPS to run from a job on the QA Jenkins box.  Woot!

Bad news:  The tests do not appear to be running correctly.  :markh, could you have a look at the above log attachment and tell me if anything jumps out at you?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The clock on this machine is diverged a lot. So make sure you have NTP installed and running. See the failure:

1456952781036	Hawk	DEBUG	(Response) /account/login?keys=true: code: 400 - Status text: Bad Request
1456952781036	Hawk	DEBUG	Clock offset vs https://api.accounts.firefox.com/v1: -1036

Comment 10

[Mark Hammond [:markh] [:mhammond]]

> CROSSWEAVE INFO: Login user: __FX_ACCOUNT_USERNAME__

It looks like config.json doesn't have the username and password of a real Firefox account - that's the default value that we expect to be edited - https://dxr.mozilla.org/mozilla-central/source/testing/tps/config/config.json.in#8

It kinda sucks that we need that, but a solution isn't obvious - I doubt we want to create a new one each run.

Comment 11

[Karl Thiessen [:kthiessen, he/him]]

Attachment: Jenkins output, 2016-03-02

Account credentials properly in place, but some unexpected failures.

Comment 12

[Karl Thiessen [:kthiessen, he/him]]

This looks much better, but I may need some help chasing down those last few failures.

Comment 13

[Mark Hammond [:markh] [:mhammond]]

> JavaScript error: resource://tps/tps.jsm, line 616: TypeError: Async.isShutdownException is not a function

my first speculation is that this is being called after Firefox has torn down, but the logs don't make it obvious that's the case. I guess I should see if I can reproduce this on Linux.

Comment 14

[Mark Hammond [:markh] [:mhammond]]

Thom, do you think you would be able to take a look at this? "tps" is a test framework that is mostly in-tree, but is a bit funky to get running. What's different about this test suite is that it is able to start and stop Firefox multiple times and with multiple profiles, and it compares the sync state between runs to check things have worked as expected. Once we get this working reliably in automation, I think it will make sense to hook your validator into it, so every test does the validation check and fails if it finds a problem. It shouldn't (as the tests run in a fairly controlled fashion, so are unlikely to trigger the cases where we screw up, such as premature shutdowns or creating bookmarks while syncing) but it still seems worthwhile - it's a way to test Sync in a way that the rest of the in-tree tests aren't.

Anyway, short term task is to get it running on Linux without errors, so it can run automagically and we get to deal with any failures.

Basic docs are at https://developer.mozilla.org/en-US/docs/Mozilla/Projects/TPS and https://developer.mozilla.org/en-US/docs/Mozilla/Projects/TPS_Tests and source is split a little between services/sync/tests/tps (the test definitions) services/sync/tps (an addon used by the harness) and testing/tps (the harness itself)

Comment 15

[Thom Chiovoloni [:tcsc] (ex-moco)]

Yep, definitely can add this to my list of things to do.

Comment 16

[Karl Thiessen [:kthiessen, he/him]]

This test is running on the QA Jenkins instance, with output going to IRC #services-test once an hour.

It's currently failing with the same error given above -- console output is here:
   https://s3-us-west-1.amazonaws.com/services-qa-jenkins-artifacts/jobs/sync_e2e-test_prod/342/test_log.txt

The '342' in that URL can be replaced by higher numbers to get output from later runs.

The QA Jenkins requires VPN access -- this job is at:
   https://services-qa-jenkins.stage.mozaws.net:8443/job/sync_e2e-test_prod/

Comment 17

[Karl Thiessen [:kthiessen, he/him]]

Many thanks to :rpapa for his tireless work getting this stood up.  Yay, Richard!

Comment 18

[Karl Thiessen [:kthiessen, he/him]]

In discussion in Sync meetings last week, Mark suggested that we break this into two pieces:

1. A minimal once-a-day job that runs against production, in such a away as not to disrupt production FxA.

2. A more thorough job that runs more often against stage.

Unless someone objects this week, I intend to assign myself these two tasks as new bugs and close this one out.

Comment 19

[Karl Thiessen [:kthiessen, he/him]]

Seems I neglected to close this out.  Doing so now: bug 1287365 is still outstanding.