Closed Bug 1045972 Opened 10 years ago Closed 9 years ago

Investigate tests for quick verification of Loop-Server deploys to Production

Categories

(Hello (Loop) :: Server, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1097244

People

(Reporter: jbonacci, Unassigned)

References

Details

(Whiteboard: [qa+][loop-fn-test])

Attachments

(1 file)

Moved the MSISDN-Gateway version of this bug over to the right category.

Opened this one even though I am sure we already did the work?
If we got it, let's document it.
If we don't have it yet, let's create it.
Whiteboard: [qa+]
Firefox Hello on B2G let you do that.
We can make you a script that let you enter your phone number and then the code so that you can make a full smoke test using your phone number.
Blocks: 1045964
That would be helpful
Moving requirement over from bug 1045964 - we need more than just a quick functional test, we also need to make sure the the Stage/Prod configurations are correct before the deployment happens.
REF: see bug 1045706
> We also need to make sure the the Stage/Prod configurations are correct before the deployment happens.

What do you mean? What kind of code do you need us to do for that?
James, that makes sense to have something that verifies the configuration; I would think a real functional test (a script making a real call between two parts) would suffice here.

If that doesn't suffice, then I would be very happy to hear what are the proposals you have in mind!

Cheers
:alexis that sounds fine with me
Great, then I think this is a dupe with bug 972025 (or maybe just a dependency of it?)
Probably a dependency.
The client team has very different goals than Services QA.
We really just want a quick way to verify that the deploy to Production was successful
(before too much regular traffic hits it)

Minimal setup, maximum feedback, quick test....
Depends on: 972025
Whiteboard: [qa+] → [qa+][loop-fn-test]
I don't think bug 972025 is necessary.  Which is to say that I think it wouldn't be hard to build on top of the code that's already in the tree so that one could, with an up-to-date Mozilla-central pull and build, one could type something like:

env LOOP_SERVER=http://loop-production-server.domain.com mach marionette-test browser/components/loop/test/functional/test_1_browser_call.py

What would be required would be to add some code to server_setup.py and config.py in 

<https://github.com/mozilla/gecko-dev/tree/master/browser/components/loop/test/functional>

to notice LOOP_SERVER environment variables that are in URL form, and to instead of a spinning up a local server, use the passed-in URL instead of localhost.

Note that we will need to be sure that this propagates through to the other bugs currently blocking 1029183, so that when they get stub server implementations of various sorts, we ensure that the live (simplepush, fxa, tokbox) servers get used.
No longer depends on: 972025
QA Contact: rpappalardo
Dan any update on this?

We are currently using our loadtest as a smoketest to validate that the server is correctly deployed.
This doesn't mean we can place an end-to-end call.

Can you elaborate on the steps we need to run your end-to-end functional test?

Thanks.
Flags: needinfo?(dmose)
Flags: needinfo?(dmose)
Nils and mreavy are driving this work now; I've redirected the needinfos to them.
Flags: needinfo?(mreavy)
Flags: needinfo?(drno)
Flags: needinfo?(mreavy)
Any news about that? Can we close? Is it tracked somewhere else? Do we need to do something on the server side to help running tests?
Flags: needinfo?(mreavy)
Flags: needinfo?(dmose)
Anthony, Nils -- What's the best bug for tracking functional tests for the client?  I believe it's Bug 1029183.  I know Nils is talking with Richard, but I don't believe there is additional server work needed for client functional testing at this time.  Please correct me if I'm wrong.

FYI: Until this is ready, the server ops folks will need help verifying that a new server deployment works with the client.  Standard8 and I have been helping with that verification, but we could use more help from QA going forward.
Flags: needinfo?(mreavy)
Flags: needinfo?(dmose)
Flags: needinfo?(anthony.s.hughes)
As far as I'm aware, to use this on non-local deploys, someone needs to write the code I described in comment 10 of this bug.  I don't believe that work is covered by any other bug, so I suspect it's best tracked here.
(In reply to Maire Reavy [:mreavy] (Plz needinfo me) from comment #14)
> Anthony, Nils -- What's the best bug for tracking functional tests for the
> client?  I believe it's Bug 1029183.  

I believe that bug should be fine but I'll let Nils/Richard confirm since they are doing most of the E2E work.
Flags: needinfo?(anthony.s.hughes) → needinfo?(rpappalardo)
Nils and I discussed this today.  For now lets go ahead and use Bug 1029183 as a kind of meta for all the e2e work/bugs.
Flags: needinfo?(rpappalardo)
Flags: needinfo?(drno)
This is an initial patch which allows to specify a HTTP URL as LOOP_SERVER env variable instead of a file location. Unfortunately I can't test it as the test itself is broken right now and I don't really have any loop server environment to point it at.

It is not clear to me yet if the whole CONTENT_SERVER thing is still needed if the test itself no longer starts the servers locally.
Presumably, the most desirable thing to do is test the content server pointed to by the loop server at the given HTTP URL.  If that's true, you're correct that we shouldn't bother starting the local content server in that case.
(In reply to Dan Mosedale (:dmose) - not reading bugmail; needinfo? for response from comment #19)
> Presumably, the most desirable thing to do is test the content server
> pointed to by the loop server at the given HTTP URL.  If that's true, you're
> correct that we shouldn't bother starting the local content server in that
> case.

Yeah I'm already not starting the Loop server and the content server. The content should basically come from the URL generated by the Loop server, correct?
So that would mean the CONTENT_SERVER related variables are hopefully not needed, and even more important we also don't need to feed them into the scripts as a second URL (additionally to the Loop server URL).
(In reply to Nils Ohlmeier [:drno] from comment #20)
>
> Yeah I'm already not starting the Loop server and the content server. The
> content should basically come from the URL generated by the Loop server,
> correct?

Right.

> So that would mean the CONTENT_SERVER related variables are hopefully not
> needed, and even more important we also don't need to feed them into the
> scripts as a second URL (additionally to the Loop server URL).

They shouldn't be needed, though it's possible that the code currently expects them.  If so, I'd think that getting rid of that dependency should be both easy and harmless.
I feel this is falling in the client part of loop, since that's who's driving the testing, and has no dependencies to the server component. Is it okay if I change the component to client ?
Flags: needinfo?(drno)
(In reply to Alexis Metaireau (:alexis) from comment #22)
> I feel this is falling in the client part of loop, since that's who's
> driving the testing, and has no dependencies to the server component. Is it
> okay if I change the component to client ?

Spoke with Nils about this.  There are already several bugs in place for developing e2e tests for loop including for functional testing of loop-server driven from the client side.
So I'm going to dupe this for Bug 1097244
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(drno)
Resolution: --- → DUPLICATE
hey nils, just wanted to make sure that we don't want the patch that was on this bug when it was duped to bug 1097244.  i don't see any review requests - so potentially not.
Flags: needinfo?(drno)
We could re-open this later (the code it not lost) if it turns out that the path with bug 10972244 is too slow or complicated.
Flags: needinfo?(drno)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: