1045972 - Investigate tests for quick verification of Loop-Server deploys to Production

Status: RESOLVED DUPLICATE of bug 1097244

(Hello (Loop) :: Server, defect)

Description

[James Bonacci [:jbonacci]]

Moved the MSISDN-Gateway version of this bug over to the right category.

Opened this one even though I am sure we already did the work?
If we got it, let's document it.
If we don't have it yet, let's create it.

Comment 1

[Rémy Hubscher (:natim)]

Firefox Hello on B2G let you do that.
We can make you a script that let you enter your phone number and then the code so that you can make a full smoke test using your phone number.

Comment 2

[James Bonacci [:jbonacci]]

That would be helpful

Comment 4

[James Bonacci [:jbonacci]]

Moving requirement over from bug 1045964 - we need more than just a quick functional test, we also need to make sure the the Stage/Prod configurations are correct before the deployment happens.
REF: see bug 1045706

Comment 5

[Rémy Hubscher (:natim)]

> We also need to make sure the the Stage/Prod configurations are correct before the deployment happens.

What do you mean? What kind of code do you need us to do for that?

Comment 6

[Alexis Metaireau (:alexis)]

James, that makes sense to have something that verifies the configuration; I would think a real functional test (a script making a real call between two parts) would suffice here.

If that doesn't suffice, then I would be very happy to hear what are the proposals you have in mind!

Cheers

Comment 7

[James Bonacci [:jbonacci]]

:alexis that sounds fine with me

Comment 8

[Alexis Metaireau (:alexis)]

Great, then I think this is a dupe with bug 972025 (or maybe just a dependency of it?)

Comment 9

[James Bonacci [:jbonacci]]

Probably a dependency.
The client team has very different goals than Services QA.
We really just want a quick way to verify that the deploy to Production was successful
(before too much regular traffic hits it)

Minimal setup, maximum feedback, quick test....

Comment 10

[Dan Mosedale (:dmosedale, :dmose)]

I don't think bug 972025 is necessary.  Which is to say that I think it wouldn't be hard to build on top of the code that's already in the tree so that one could, with an up-to-date Mozilla-central pull and build, one could type something like:

env LOOP_SERVER=http://loop-production-server.domain.com mach marionette-test browser/components/loop/test/functional/test_1_browser_call.py

What would be required would be to add some code to server_setup.py and config.py in 

<https://github.com/mozilla/gecko-dev/tree/master/browser/components/loop/test/functional>

to notice LOOP_SERVER environment variables that are in URL form, and to instead of a spinning up a local server, use the passed-in URL instead of localhost.

Note that we will need to be sure that this propagates through to the other bugs currently blocking 1029183, so that when they get stub server implementations of various sorts, we ensure that the live (simplepush, fxa, tokbox) servers get used.

Comment 11

[Rémy Hubscher (:natim)]

Dan any update on this?

We are currently using our loadtest as a smoketest to validate that the server is correctly deployed.
This doesn't mean we can place an end-to-end call.

Can you elaborate on the steps we need to run your end-to-end functional test?

Thanks.

Comment 12

[Dan Mosedale (:dmosedale, :dmose)]

Nils and mreavy are driving this work now; I've redirected the needinfos to them.

Comment 13

[Rémy Hubscher (:natim)]

Any news about that? Can we close? Is it tracked somewhere else? Do we need to do something on the server side to help running tests?

Comment 14

[Maire Reavy [:mreavy]]

Anthony, Nils -- What's the best bug for tracking functional tests for the client?  I believe it's Bug 1029183.  I know Nils is talking with Richard, but I don't believe there is additional server work needed for client functional testing at this time.  Please correct me if I'm wrong.

FYI: Until this is ready, the server ops folks will need help verifying that a new server deployment works with the client.  Standard8 and I have been helping with that verification, but we could use more help from QA going forward.

Comment 15

[Dan Mosedale (:dmosedale, :dmose)]

As far as I'm aware, to use this on non-local deploys, someone needs to write the code I described in comment 10 of this bug.  I don't believe that work is covered by any other bug, so I suspect it's best tracked here.

Comment 16

[u279076]

(In reply to Maire Reavy [:mreavy] (Plz needinfo me) from comment #14)
> Anthony, Nils -- What's the best bug for tracking functional tests for the
> client?  I believe it's Bug 1029183.  

I believe that bug should be fine but I'll let Nils/Richard confirm since they are doing most of the E2E work.

Comment 17

[Richard Pappalardo [:rpapa][:rpappalardo]]

Nils and I discussed this today.  For now lets go ahead and use Bug 1029183 as a kind of meta for all the e2e work/bugs.

Comment 18

[Nils Ohlmeier [:drno]]

Attachment: bug_1045972_e2e_test_on_server.patch

This is an initial patch which allows to specify a HTTP URL as LOOP_SERVER env variable instead of a file location. Unfortunately I can't test it as the test itself is broken right now and I don't really have any loop server environment to point it at.

It is not clear to me yet if the whole CONTENT_SERVER thing is still needed if the test itself no longer starts the servers locally.

Comment 19

[Dan Mosedale (:dmosedale, :dmose)]

Presumably, the most desirable thing to do is test the content server pointed to by the loop server at the given HTTP URL.  If that's true, you're correct that we shouldn't bother starting the local content server in that case.

Comment 20

[Nils Ohlmeier [:drno]]

(In reply to Dan Mosedale (:dmose) - not reading bugmail; needinfo? for response from comment #19)
> Presumably, the most desirable thing to do is test the content server
> pointed to by the loop server at the given HTTP URL.  If that's true, you're
> correct that we shouldn't bother starting the local content server in that
> case.

Yeah I'm already not starting the Loop server and the content server. The content should basically come from the URL generated by the Loop server, correct?
So that would mean the CONTENT_SERVER related variables are hopefully not needed, and even more important we also don't need to feed them into the scripts as a second URL (additionally to the Loop server URL).

Comment 21

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to Nils Ohlmeier [:drno] from comment #20)
>
> Yeah I'm already not starting the Loop server and the content server. The
> content should basically come from the URL generated by the Loop server,
> correct?

Right.

> So that would mean the CONTENT_SERVER related variables are hopefully not
> needed, and even more important we also don't need to feed them into the
> scripts as a second URL (additionally to the Loop server URL).

They shouldn't be needed, though it's possible that the code currently expects them.  If so, I'd think that getting rid of that dependency should be both easy and harmless.

Comment 22

[Alexis Metaireau (:alexis)]

I feel this is falling in the client part of loop, since that's who's driving the testing, and has no dependencies to the server component. Is it okay if I change the component to client ?

Comment 23

[Richard Pappalardo [:rpapa][:rpappalardo]]

(In reply to Alexis Metaireau (:alexis) from comment #22)
> I feel this is falling in the client part of loop, since that's who's
> driving the testing, and has no dependencies to the server component. Is it
> okay if I change the component to client ?

Spoke with Nils about this.  There are already several bugs in place for developing e2e tests for loop including for functional testing of loop-server driven from the client side.
So I'm going to dupe this for Bug 1097244

Comment 24

[:shell escalante]

hey nils, just wanted to make sure that we don't want the patch that was on this bug when it was duped to bug 1097244.  i don't see any review requests - so potentially not.

Comment 25

[Nils Ohlmeier [:drno]]

We could re-open this later (the code it not lost) if it turns out that the path with bug 10972244 is too slow or complicated.