902694 - Need 4 AWS VMs for B2G emulator memory tests

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jonathan Griffin (:jgriffin)]

Per jlebar's request, Rob Wood is going to set up some B2G memory tests that will be run on B2G emulators and report to areweslimyet.com (AWSY).  To run these, we'd like a set of 4 AWS VMs with the current Ubuntu64 test slave configuration, but not inside the build VPN.

One of these VMs, which will host a Jenkins master, should be open to the internet on port 8080; the others don't need any public open ports, but should be able to accept ssh connections from the master.

Is this something you guys in releng can set up for us?  It's something like the VMs you used to provide us for B2G unit testing, before all such testing was moved to buildbot/TBPL.

Comment 1

[Justin Lebar (not reading bugmail)]

Please contact dougt or jst if you need approvals here.

Comment 2

[Justin Wood (:Callek)]

Ok, I may have moved this into the Loaners component prematurely..

I read this initially as "We want to get 4 AWS Ubuntu VMs like you use for current tests"

On re-reading this it sounds like you want 4 near-permanent machines to run a set of tests over jenkins that Mozilla Releng will not maintain and report elsewhere than tbpl/etc.

If you do only want loaners I can get them to you pretty fast (will have an outstanding Q or two about the ports open to the internet though) if its actually my new understanding and permanent I'll need to have a chat within releng to find out what we can recommend as a proper path forward here.

sending needinfo to all names on this bug so far, to expedite an answer (in an attempt to not block you guys for longer than necessary)

Comment 3

[Justin Lebar (not reading bugmail)]

> On re-reading this it sounds like you want 4 near-permanent machines to run a set of tests over 
> jenkins that Mozilla Releng will not maintain and report elsewhere than tbpl/etc.

That's what I understand jgriffin is asking for.

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Justin Lebar [:jlebar] (limited availability 8/9 – 8/12) from comment #3)
> > On re-reading this it sounds like you want 4 near-permanent machines to run a set of tests over 
> > jenkins that Mozilla Releng will not maintain and report elsewhere than tbpl/etc.
> 
> That's what I understand jgriffin is asking for.

Yes, that's what we'd like; specifically, we are not asking for temporary loaners.

Comment 5

[cmtalbert]

The reason this is in the releng request queue is because we are going to run these memory tests per-checkin on b2g emulators. Because of that load, because we want to run on exactly the same VM configuration that the emulator tests currently run on (and for these machines to remain current as those machines are updated), and because of the various restrictions we always hit moving b2g builds around, we would ideally want these to live in the releng VPC since it is already configured to handle these issues.

And that's why this request is here.

Comment 6

[cmtalbert]

Sorry for the spam, per-checkin meaning - they would run constantly, but they would likely run for long periods of time.  So the load would be high, but they would *not* be able to run every single checkin coming through the scheduler, hence the reason for their own scheduler.

Comment 7

[Jonathan Griffin (:jgriffin)]

Catlee, is this something you can help out with?  I know you've managed some of our AWS VM's in the past.

Comment 8

[Chris AtLee [:catlee]]

So it sounds like the ask is for 4 VMs inside RelEng's VPC, and to have one of these instances be accessibly publicly so it can act as a jenkins server?

I don't see a problem setting up the test workers inside the VPC, but setting up a publicly accessible jenkins server worries me. Is there another place we can put this?

Comment 9

[Jonathan Griffin (:jgriffin)]

We could hook it up to the existing Jenkins master at http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per discussion with jgriffin in b2g meeting just now:

1) it would be helpful if RelEng were able to create these 4 instances, like typical loaner test slaves, allow the new instances to connect to puppet for the initial setup, then scrub production keys, power off, move outside of the RelEng VPC, and bring back up. Lets try this and see if it works.
1a) These machines would no longer be able to connect to RelEng  puppetmaster, so over time would become out of sync with current  production test machines. However, jgriffin is ok with this for now. If  these are *so* out of sync that it becomes a problem, we can delete and  recreate again.
1b) If that isnt possible, worst case, jgriffin could create these 4 instances as just "plain" instances, in which case jgriffin will move this bug to Testing:General and then create the 4 instances in the ateam AWS account. However, RelEng's initial setup in (1) would be preferred.

2) For running, these 4 VMs do not need to run in the RelEng VPC. They will be downloading builds from public ftp.m.o, cloning from public hg.m.o, and need to communicate with  http://ec2-107-20-108-245.compute-1.amazonaws.com/jenkins/, which means they need to be accessible via ssh *from* that machine.

Because of all this, moving this bug back into buildduty component.

Comment 11

[Chris AtLee [:catlee]]

It's not possible to move instances in/out of VPCs.

Comment 12

[Jonathan Griffin (:jgriffin)]

All right, then it looks like we'll need to create 4 generic Ubuntu VM's unrelated to the test slaves.  I can set up a VM locally on my account and then have it cloned in either the rel-eng or a-team AWS accounts.

Comment 13

[Chris AtLee [:catlee]]

joduinn and I spoke about this yesterday, and I think I understand the requirements better.

If we can get you an AMI for one of our test machine instances, would that be useful? Then you can create whatever instances you need in your own account.

Comment 14

[Jonathan Griffin (:jgriffin)]

Sure, that would be fine.  Thanks!

Comment 15

[Chris AtLee [:catlee]]

Ok, here's what I've done:

- Stopped an existing instance (tst-linux64-ec2-302) to create an AMI from it
- Create a new instance from that AMI, boot it up
- Cleanup:
  - remove /builds/slave
  - remove /var/lib/puppet/
  - remove ~cltbld/.vnc/passwd
  - set up ssh key management (/etc/rc.local and /usr/local/bin/get-credentials.sh from https://forums.aws.amazon.com/message.jspa?messageID=76866#)
  - truncate /root/.ssh/authorized_keys

- Created an AMI from this scrubbed instance. The AMI id is ami-24188414. I need your account # to share it with you.

Comment 16

[Jonathan Griffin (:jgriffin)]

glob, can you e-mail catlee our AMS account id, so we can clone the AMI he's set up?

Comment 17

[:glob ✱]

emailed.

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage. catlee, now that you're back from PTO, whats status here?

Comment 19

[:glob ✱]

my bad -- catlee provided me with the details but i haven't been able to see the ami, and i failed to ask catlee for assistance.  i'll work with him over email to avoid bugspam.

Comment 20

[cmtalbert]

(In reply to Byron Jones ‹:glob› from comment #19)
> my bad -- catlee provided me with the details but i haven't been able to see
> the ami, and i failed to ask catlee for assistance.  i'll work with him over
> email to avoid bugspam.

Thanks Glob, let rwood know as soon as you have the machines up so he can start testing with them.

Comment 21

[Chris AtLee [:catlee]]

glob confirmed on irc last week he is able to see the AMI. If you have any problems with it, let me know!

Comment 22

[Robert Wood [:rwood]]

:glob ping, just wondering if you have any updates on the VMs? Thanks :)

Comment 23

[:glob ✱]

(In reply to Rob Wood [:rwood] from comment #22)
> :glob ping, just wondering if you have any updates on the VMs? Thanks :)

worked a bit on it today, hit an issue with ssh keys.  i'll work through this with catlee via email/irc.

Comment 24

[:glob ✱]

created:

ec2-54-200-198-57.us-west-2.compute.amazonaws.com
ec2-54-200-198-58.us-west-2.compute.amazonaws.com
ec2-54-200-193-154.us-west-2.compute.amazonaws.com
ec2-54-200-194-98.us-west-2.compute.amazonaws.com

rwood has the connections details has is able to connect.