926300 - Create additional VMs for each supported platform (Linux and Windows) in qa.scl3.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Now that bug 922080 has been fixed we would like to request a set of new VMs for our Mozmill-CI instance. This is necessary so we can make sure to test beta and releases for all locales of Firefox.

That means we need:
1x Ubuntu 12.04 32
1x Ubuntu 12.04 64
1x Ubuntu 13.04 32
1x Ubuntu 13.04 64
1x Windows XP
1x Windows Vista
1x Windows 7 32
1x Windows 7 64
1x Windows 8 32
1x Windows 8 64

Thanks

Comment 1

[Rick Bryce [:rbryce]]

We will need to know what server specs you will need for each of these servers.  As well the virtualization team may want to handle this vm deployment load in a particular way.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Rick, sorry for filing it in the wrong component. So we have templates for all of those VMs. Adrian knows about them.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

This is currently blocked on bug 916724, given that the Linux machines have wrong proxy settings and need an update.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Adrian, I think we should also wait for bug 880709, which I will be able to hopefully later today. Sorry, but that would require another update of all templates then. Thankfully we got the ACL fixed by netops today.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Adrian, I think we shouldn't wait for bug 880709 to be fixed. I will get to that sometime that week, but the 4th node has a way higher priority for us. Also I can do the remaining update for those like I have to do for all the other nodes.

So whenever you have a free slot please continue and create all those additional nodes. Thanks

Comment 6

[Adrian J Fernandez [:Aj]]

Currently at a work-week, so expect some delays but should get done by tomorrow (or possibly today).

Comment 7

[Adrian J Fernandez [:Aj]]

The requested VMs are online, added to inventory and documentation updated.

mm-ub-1204-32-4.qa.scl3.mozilla.com
mm-ub-1204-64-4.qa.scl3.mozilla.com
mm-ub-1304-32-4.qa.scl3.mozilla.com
mm-ub-1304-64-4.qa.scl3.mozilla.com
mm-win-xp-32-4.qa.scl3.mozilla.com
mm-win-vista-32-4.qa.scl3.mozilla.com
mm-win-7-32-4.qa.scl3.mozilla.com
mm-win-7-64-4.qa.scl3.mozilla.com already online (4th), unless you want a 5th one.
mm-win-8-32-4.qa.scl3.mozilla.com
mm-win-8-64-4.qa.scl3.mozilla.com

Windows VMs MAY need a license to activate so please re-open if you do see the warnings (no warnings yet).

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

(In reply to Adrian Fernandez [:Aj] from comment #7)
> mm-ub-1304-32-4.qa.scl3.mozilla.com
> mm-win-xp-32-4.qa.scl3.mozilla.com
> mm-win-vista-32-4.qa.scl3.mozilla.com
> mm-win-7-32-4.qa.scl3.mozilla.com
> mm-win-8-32-4.qa.scl3.mozilla.com
> mm-win-8-64-4.qa.scl3.mozilla.com

All of those VMs have been connected to Jenkins successfully.

> mm-ub-1204-32-4.qa.scl3.mozilla.com
> mm-ub-1304-64-4.qa.scl3.mozilla.com

I had to reboot both machines given that VNC didn't work. Now it works but they have problems with connecting to ci-master due to the proxy. Not sure what is causing this and how this is related. I will have to investigate.

> mm-ub-1204-64-4.qa.scl3.mozilla.com

Here I can't connect via VNC at all. You will have to check that Adrian. I cannot find the culprit via the command line.

> mm-win-7-64-4.qa.scl3.mozilla.com already online (4th), unless you want a
> 5th one.

Not sure why that already existed. I have never requested one. Can you please delete it and create a new one? All settings in this VM will be completely outdated. Thanks.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Adrian, I was wondering earlier today already and now it's clear. The templates you have used are not the ones which I have updated recently. At least the 12.04 32bit one does not include the changes made by bug 911795.

Can you please check that?

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

> mm-ub-1304-64-4.qa.scl3.mozilla.com

That one is online now.

> mm-ub-1204-32-4.qa.scl3.mozilla.com

Michael, I have huge issues in getting Firefox to connect to our http://mm-ci-master.qa.scl3.mozilla.com:8080 website. Firefox is always trying to use the proxy for that which will fail with Access denied. I have absolutely no idea why Firefox is not taking the no_proxy or ignore_list from the system. Do you have any idea?

Comment 11

[Adrian J Fernandez [:Aj]]

Henrik, do you want me to convert the 12.04 32 bit template back to a VM?
I am checking the VM task/events log and the VMs were indeed all converted back to templates on September 5th (as per the last bug comment in Bug 911795). Perhaps your changes were missed on this one?

As for Firefox not working with the proxy, did you re-apply the changes that were previously supposed to be made and see if it works now?

Comment 12

[Adrian J Fernandez [:Aj]]

Forgot to add, 

mm-win-7-64-4.qa.scl3.mozilla.com will be re-deployed.
mm-ub-1204-64-4.qa.scl3.mozilla.com was at login prompt, I logged in and rebooted the node and VNC is working.

If there's anything else that needs to be checked (minus the proxy), please let me know.

Comment 13

[Adrian J Fernandez [:Aj]]

mm-win-7-64-4.qa.scl3.mozilla.com back online.

Leaving bug open until proxy issue could be resolved but if that's indeed an issue, it should be a separate bug.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

(In reply to Adrian Fernandez [:Aj] from comment #11)
> Henrik, do you want me to convert the 12.04 32 bit template back to a VM?

I think that would be good. So I could check what's wrong there.

> I am checking the VM task/events log and the VMs were indeed all converted
> back to templates on September 5th (as per the last bug comment in Bug
> 911795). Perhaps your changes were missed on this one?

Those were 4 VMs and I clearly did everything sequentially to not miss something. So I really don't understand this. :(

> As for Firefox not working with the proxy, did you re-apply the changes that
> were previously supposed to be made and see if it works now?

All the proxy settings are totally equal to the other boxes, but Firefox is still not able to reach our jenkins website.

I will work on the integration of the other VMs now.

Comment 15

[Adrian J Fernandez [:Aj]]

As per yesterday, issue is not with the proxy but a local issue.
mm-ub-1204-32-4.qa.scl3.mozilla.com
mm-ub-1304-64-4.qa.scl3.mozilla.com

Can connect now. I know we did some troubleshooting but my work-week got the best of me.
Did you determine culprit?

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Actually not, given that I was waiting for your reply. :) You are most likely not around now, so I will do further checks regarding the /etc/resolve.conf differences we have seen.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1] (away from Feb 12th - Feb 22nd)]

Looks like the command which we missed to execute is:

gsettings set org.gnome.system.proxy ignore-hosts "['localhost','127.0.0.0/8','10.0.0.0/8','*.scl3.mozilla.com','*.phx1.mozilla.com']"

I also updated the tail file for resolve.conf.

Adrian, can you please check that also for the templates? It should be wrong there too. Also it would be great when we get our Mana documentation updated, so it specifies what we really have to do. As of now it looks kinda nontransparent. 

I would suggest that we do all the proxy config settings via the terminal. As best via a script, which we would need for Puppet anyway, right?

Comment 18

[Adrian J Fernandez [:Aj]]

Following VM templates updated;
Ubuntu_12.04_x86
Ubuntu_13.04_x86-64

As for the documentation, it does list; gsettings set org.gnome.system.proxy ignore-hosts etc
so perhaps an update (apt-get etc) blew the changes.

As for what needs to be done, pretty much everything that's listed in the proxy side of things need to be done on any NEW template we are creating.