1107286 - Make marketplace test results public

Status: VERIFIED FIXED

(Marketplace Graveyard :: General, defect, P1)

Description

[krupa raj[:krupa]]

We have a lot of tests which run on marketplace which are currently behind the Mtn.View VPN. We need to make it available widely. Assigning to Stephen since he is interested in getting this done as well.

Comment 1

[Kumar McMillan [:kumar]]

We are using https://travis-ci.org/ now for most projects so that ops doesn't have to maintain a custom CI server. Is it possible to run the tests there?

Comment 2

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Kumar McMillan [:kumar] (needinfo for quickness) from comment #1)
> We are using https://travis-ci.org/ now for most projects so that ops
> doesn't have to maintain a custom CI server. Is it possible to run the tests
> there?

I'm for that, but there's a limitation, per http://docs.travis-ci.com/user/sauce-connect/:

"Note that due to security restrictions, the Sauce Labs addon is not available on pull request builds. See the pull requests page for full details on why it is disabled." -- we need Sauce Labs because we don't have the manpower to build out and maintain an Android lab, so that's a consideration in this.

Bob, can you please take a look at how far along using Travis we can get with the AMO + Marketplace suites, and what the ramifications of using or not using it or, to help mitigate this visibility issue?  Thanks!

Comment 3

[Bob Silverberg [:bsilverberg]]

Stephen, I know that we've discussed the possibility of opening up our CI to those outside of the VPN. Would that be a solution to this issue, or do you feel that we should also look to Travis even if we can open up our CI?

Comment 4

[Bob Silverberg [:bsilverberg]]

As a follow-up to this, I have done an experiment with the technique for eliminating the possible credentials exposure on our CI, and it seems to work well.

If you look at [1], you'll see:

self = <browserid.browser_id.BrowserID object at 0x109cbda90>, email = '<value>'
password = '<value>'

in the stack trace. That '<value>' is the actual value from the credentials file, and hence would be exposed if it were a real username and password.

If you look at [2], you'll see that the stack trace omits those function arguments and therefore cannot expose the credentials.

If we would like to, we can implement this solution which would allow us to open up our CI to the world.


[1] http://selenium.qa.mtv2.mozilla.com:8080/job/moztrap.dev.expose/1/HTML_Report/
[2] http://selenium.qa.mtv2.mozilla.com:8080/view/MozTrap/job/moztrap.dev.no_expose/1/HTML_Report/

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Bob Silverberg [:bsilverberg] from comment #3)
> Stephen, I know that we've discussed the possibility of opening up our CI to
> those outside of the VPN. Would that be a solution to this issue, or do you
> feel that we should also look to Travis even if we can open up our CI?

That would be great; Dave also has looked into that, and I believe has a solution we can and should try.

Another point Dave raised is that it might be a good idea to get a public-facing Jenkins instance (virtual machine in SCL3 -- so it could then also talk to the Selenium Node Windows VM hosts we have in SCL3), where we could start to transition projects over, without losing (much) time; I anticipate that it might be easier (and it's better to have it be a virtual machine for capacity, reliability, etc.), than opening up netflows in the current Mountain View QA lab to and from our current Mac-Mini-hosted Jenkins.

Comment 6

[Bob Silverberg [:bsilverberg]]

That sounds like a good idea, Stephen. @Davehunt, do you know who set up the Jenkins instance for b2g? I am wondering if that person could either help get another one up for us, or provide some guidance to me to get it done.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Raymond was the driving force behind the Jenkins instance for B2G. I found the original request for the instance in bug 973098, and there are a few related dependencies indicated in bug 1019792.

I would suggest we file a bug requesting:

Hostname: webqa-ci1
Specification: RHEL 6, 16GB RAM, 4 cores, 250G HDD

This would match the instance currently used for B2G. We should have the latest Jenkins LTS version installed on this box. The Web QA team should have SSH access to this box, and will ideally have write access to the Jenkins home path. As much of this that can be automated via puppet or similar would be great. We should also have monitoring for this box, and ideally a backup solution for at least the Jenkins home path.

Finally, the intention is that the instance is publicly accessible. Hopefully this can be from the outset, however I suspect this adds some complications.

Comment 8

[Bob Silverberg [:bsilverberg]]

Thanks for all the info, Dave. I will open a new bug based on bug 973098 to create the new instance and we can proceed from there. I don't currently have access to that bug, so if you could cc me on it that would be great.

Stephen, this still sounds like a fairly major undertaking. If time is truly of the essence it might be quicker to try opening up our existing Jenkins instance. What do you think?

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Bob Silverberg [:bsilverberg] from comment #8)
> Thanks for all the info, Dave. I will open a new bug based on bug 973098 to
> create the new instance and we can proceed from there. I don't currently
> have access to that bug, so if you could cc me on it that would be great.
> 
> Stephen, this still sounds like a fairly major undertaking. If time is truly
> of the essence it might be quicker to try opening up our existing Jenkins
> instance. What do you think?

Yes, Bob, sounds like a good approach to me.  http://webqa-ci1.qa.scl3.mozilla.com:8080/ is still behind VPN, but we can soon hopefully open that up with the right safeguards in place.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

There are a few ways to solve this, moving to ci.mozilla.org is just one (and possibly not the preferred choice). Making this a tracking bug.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

All done: https://webqa-ci.mozilla.com/view/Marketplace/

Comment 12

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Dave Hunt (:davehunt) from comment #11)
> All done: https://webqa-ci.mozilla.com/view/Marketplace/

\O/ - Thanks, Dave! <3