Closed
Bug 772579
Opened 12 years ago
Closed 11 years ago
evaluate RelEng AMI for running linux desktop unittests
Categories
(Testing :: General, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: joduinn, Assigned: jmaher)
References
Details
from irc: 1) last week (or before?), rail gave jmaher an AMI so jmaher could try to get linux desktop unittests on it. The idea here is to see 1a) if linux desktop unittests can run on an AMI 1b) if unittests on AMI are as reliable/consistent as unittests on physical hardware (historically we had intermittent orange noise when running unittests on our ESX VMs). 2) jmaher has everything he needs, but reports some problem with the X not running on the AMI? 3) No bug filed to track this project yet, so filing as requested.
Assignee | ||
Comment 1•12 years ago
|
||
access is good; need to allocate 4 or 5 more hours of heads down on this to get firefox going.
Assignee | ||
Comment 2•12 years ago
|
||
I have mochitests running on the ec2 box. Next steps: * verify all tests run * verify all tests pass * create setup steps on the AMI machine to replicate this * look at failure rates and ensure we at parity if not better than orange rates on our current production machines * regroup
Assignee | ||
Comment 3•12 years ago
|
||
In order to get this running, I had to put X in a virtual frame buffer. I don't know if this is acceptable or not. reliable: jsreftests crashtests unreliable: reftest mochitest todo: mochitest-chrome browser-chrome xpcshell
Assignee | ||
Comment 4•12 years ago
|
||
I am running these tests with Xfvb, do we care if we have a virtual framebuffer. :ctalbert mentioned that we had moved away from virtual frame buffers in 2008, but he couldn't remember a reason.
One issue here is that with vms running desktop UIs we *might* have some kind of true hardware support so that the tests are running with hardware acceleration. If we have to use xvfb, then we have 0 hardware acceleration testing because our entire rendering stack is software at that point. Given that, I don't think this is going to be viable for testing of anything that uses a UI. So that means reftest and mochitest and Talos are out. :( That said, running c++ unit tests and XPCShell tests and JS Shell tests (we don't run JSShell on all branches) would work fine on these systems.
Assignee | ||
Comment 6•12 years ago
|
||
I think jsreftests would be good as well, but we would need xvfb even though we are not actively testing the ui. Currently xpcshell runs, and I get 9 failures. That is pretty consistent. releng: does it make sense to pursue this further? If so I could work on a cleaner machine setup and steps to run xpcshell and maybe jstests
Comment 7•12 years ago
|
||
do all UI tests require HW acceleration? perhaps we should ask the GFX team for input on that?
(In reply to Chris AtLee [:catlee] from comment #7) > do all UI tests require HW acceleration? perhaps we should ask the GFX team > for input on that? If you could ask while you have the entire team captive in Toronto this week, I'd be much obliged. :)
Comment 9•12 years ago
|
||
Jeff says it's perfectly fine with him :)
Comment 10•12 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #9) > Jeff says it's perfectly fine with him :) Perfectly fine running the reftests etc without hardware accleration? That's a different stance than they've taken in the past, but it would certainly make our lives easier.
Comment 11•12 years ago
|
||
(In reply to Clint Talbert ( :ctalbert ) from comment #10) > (In reply to Chris AtLee [:catlee] from comment #9) > > Jeff says it's perfectly fine with him :) > > Perfectly fine running the reftests etc without hardware accleration? That's > a different stance than they've taken in the past, but it would certainly > make our lives easier. Catlee, can you be sure he understood what we were asking here? This sounds very different from the take we've had in the past.
Comment 12•12 years ago
|
||
So, what I think is that we should definitely be able to run * jsreftest * xpcshell in the cloud. Those don't do anything with hardware acceleration. And I think those could be directed up here immediately. Are there others we could potentially put up here? Perhaps browser-chrome if it runs well under XVFB? Do we need someone to follow up with the devs on this? If so, I'll volunteer to do that.
Assignee | ||
Comment 13•12 years ago
|
||
I think we can make a case for both of these tests in the cloud to start with. If this goes well, we can consider others: * crashtests * browser-chrome * chrome I would think reftest and mochitest-plain would be hard to pull off on this virtual environment.
Reporter | ||
Comment 14•12 years ago
|
||
jmuizelaar, catlee, jmaher: After comment#9...#13, I cant tell for certain, hence the explicit ask: 1) What explicit list linux desktop unittest suites are now known to be ok to run on AWS? 2) Anything left to do here before we start migrating those unittest suites over in production?
Assignee | ||
Comment 15•12 years ago
|
||
I will be able to look at this again after the mobile work week, but right now it is looking like we can run jsreftests and xpcshell on AWS. What needs to be done is: 1) I need to replicate this on a fresh box 2) I need to document what needs to be installed 3) We need to make a master image to use 4) profit
Comment 16•12 years ago
|
||
2.5) create puppet manifests for what needs to be installed
Reporter | ||
Comment 17•12 years ago
|
||
(In reply to Joel Maher (:jmaher) from comment #15) > I will be able to look at this again after the mobile work week, but right > now it is looking like we can run jsreftests and xpcshell on AWS. > > What needs to be done is: > 1) I need to replicate this on a fresh box > 2) I need to document what needs to be installed > 3) We need to make a master image to use > 4) profit (In reply to Chris AtLee [:catlee] from comment #16) > 2.5) create puppet manifests for what needs to be installed jmaher: We'd prefer to *not* create a new AMI/master image. Instead, using the same base AMI, and using puppet to install anything else as-necessary, allows us to keep our AWS and inhouse machines in sync... so we can schedule jsreftests/xpcshell tests on *either* AWS or inhouse machines. Therefore, aiui, the summary of comment#15, comment#16 is: 1) jmaher to replicate this on a fresh box - let us know when you are ready, and catlee will create a new instance for you 2) jmaher need to document what needs to be installed 3) jmaher/catlee to create puppet manifests of any changes to AMI 4) RelEng to evaluate jsreftests and xpcshell on AWS in staging 5) RelEng to enable jsreftests and xpcshell on AWS in production 6) profit (or at least slightly better wait times!)
Comment 18•12 years ago
|
||
(In reply to John O'Duinn [:joduinn] from comment #17) > jmaher: > We'd prefer to *not* create a new AMI/master image. Instead, using the same > base AMI, and using puppet to install anything else as-necessary, allows us > to keep our AWS and inhouse machines in sync... so we can schedule > jsreftests/xpcshell tests on *either* AWS or inhouse machines. Therefore, > aiui, the summary of comment#15, comment#16 is: I think it depends how divergent the two configs are going to be and how much testing we can push to the cloud, frankly. Also, we may *need* to create a new AMI if we want to run tests on 32-bit vs. 64-bit, since AFAIK the existing base image is 64-bit.
Comment 19•12 years ago
|
||
I doubt we can use the same base AMI as we're using for the build machine as well. It's likely we'll need a new AMI for the test machines. However, we should be following the same pattern as we used for the build machines: - one AMI which represents the minimal base image with no packages installed. the creation of this AMI needs to be scripted so it can be repeated in different regions. we have tools already written to create the base build AMI, so I imagine creating a new base test AMI will be simple. - one AMI that has been puppetized with all the required packages. this AMI is used to create actual instances that will be connected to buildbot. The key points here are that the images need to be puppetized and easily reproducible in other AWS regions, or other cloud providers.
Reporter | ||
Comment 20•12 years ago
|
||
Filed bug#784913 to track work for migrating linux xpcshell testsuite to AWS. Turns out there's some unresolved questions about jsreftests, will file a separate bug to migrate once those questions are resolved by JS group.
Reporter | ||
Comment 21•11 years ago
|
||
(In reply to John O'Duinn [:joduinn] from comment #20) > Filed bug#784913 to track work for migrating linux xpcshell testsuite to > AWS. Turns out there's some unresolved questions about jsreftests, will file > a separate bug to migrate once those questions are resolved by JS group. bug#784913 and then bug#837022 tracked getting xpcshell running on AWS and that work is done. Whats left to migrate to AWS? If anything, is that remaining work already being tracked in another bug, so we can close this bug?
Comment 22•11 years ago
|
||
We have bug 837017 to track desktop unittest migration from Fedora to AWS Ubuntu. We need to move mochitests-{1..5}, mochitest-browser-chrome and mochitest-others.
Assignee | ||
Comment 23•11 years ago
|
||
we have completed this project. Remaining work for moving all tests will be in bugs: bug 850101 - browser-chrome bug 850103 - mochitest-other
Reporter | ||
Comment 24•11 years ago
|
||
(In reply to Joel Maher (:jmaher) from comment #23) > we have completed this project. Remaining work for moving all tests will be > in bugs: > bug 850101 - browser-chrome > bug 850103 - mochitest-other ..and per in-person chat, this bug can now be closed as FIXED. :-)
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•