Create additional VMs for each supported platform (Linux and Windows) in qa.scl3.mozilla.com

RESOLVED FIXED

Status

Infrastructure & Operations
Virtualization
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: whimboo, Assigned: Aj)

Tracking

Details

(Whiteboard: [qa-automation-wanted], URL)

Now that bug 922080 has been fixed we would like to request a set of new VMs for our Mozmill-CI instance. This is necessary so we can make sure to test beta and releases for all locales of Firefox.

That means we need:
1x Ubuntu 12.04 32
1x Ubuntu 12.04 64
1x Ubuntu 13.04 32
1x Ubuntu 13.04 64
1x Windows XP
1x Windows Vista
1x Windows 7 32
1x Windows 7 64
1x Windows 8 32
1x Windows 8 64

Thanks

Comment 1

4 years ago
We will need to know what server specs you will need for each of these servers.  As well the virtualization team may want to handle this vm deployment load in a particular way.
Assignee: server-ops → server-ops-virtualization
Component: Server Operations → Server Operations: Virtualization
QA Contact: shyam → dparsons
Rick, sorry for filing it in the wrong component. So we have templates for all of those VMs. Adrian knows about them.
Summary: Create additional VMs for each supported platform (Linux and Windows) → Create additional VMs for each supported platform (Linux and Windows) in qa.scl3.mozilla.com
This is currently blocked on bug 916724, given that the Linux machines have wrong proxy settings and need an update.
Depends on: 916724

Updated

4 years ago
Assignee: server-ops-virtualization → afernandez
Adrian, I think we should also wait for bug 880709, which I will be able to hopefully later today. Sorry, but that would require another update of all templates then. Thankfully we got the ACL fixed by netops today.
Depends on: 880709
Adrian, I think we shouldn't wait for bug 880709 to be fixed. I will get to that sometime that week, but the 4th node has a way higher priority for us. Also I can do the remaining update for those like I have to do for all the other nodes.

So whenever you have a free slot please continue and create all those additional nodes. Thanks
Status: NEW → ASSIGNED
No longer depends on: 880709
Flags: needinfo?(afernandez)
(Assignee)

Comment 6

4 years ago
Currently at a work-week, so expect some delays but should get done by tomorrow (or possibly today).
(Assignee)

Comment 7

4 years ago
The requested VMs are online, added to inventory and documentation updated.

mm-ub-1204-32-4.qa.scl3.mozilla.com
mm-ub-1204-64-4.qa.scl3.mozilla.com
mm-ub-1304-32-4.qa.scl3.mozilla.com
mm-ub-1304-64-4.qa.scl3.mozilla.com
mm-win-xp-32-4.qa.scl3.mozilla.com
mm-win-vista-32-4.qa.scl3.mozilla.com
mm-win-7-32-4.qa.scl3.mozilla.com
mm-win-7-64-4.qa.scl3.mozilla.com already online (4th), unless you want a 5th one.
mm-win-8-32-4.qa.scl3.mozilla.com
mm-win-8-64-4.qa.scl3.mozilla.com

Windows VMs MAY need a license to activate so please re-open if you do see the warnings (no warnings yet).
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Flags: needinfo?(afernandez)
Resolution: --- → FIXED
(In reply to Adrian Fernandez [:Aj] from comment #7)
> mm-ub-1304-32-4.qa.scl3.mozilla.com
> mm-win-xp-32-4.qa.scl3.mozilla.com
> mm-win-vista-32-4.qa.scl3.mozilla.com
> mm-win-7-32-4.qa.scl3.mozilla.com
> mm-win-8-32-4.qa.scl3.mozilla.com
> mm-win-8-64-4.qa.scl3.mozilla.com

All of those VMs have been connected to Jenkins successfully.

> mm-ub-1204-32-4.qa.scl3.mozilla.com
> mm-ub-1304-64-4.qa.scl3.mozilla.com

I had to reboot both machines given that VNC didn't work. Now it works but they have problems with connecting to ci-master due to the proxy. Not sure what is causing this and how this is related. I will have to investigate.

> mm-ub-1204-64-4.qa.scl3.mozilla.com

Here I can't connect via VNC at all. You will have to check that Adrian. I cannot find the culprit via the command line.

> mm-win-7-64-4.qa.scl3.mozilla.com already online (4th), unless you want a
> 5th one.

Not sure why that already existed. I have never requested one. Can you please delete it and create a new one? All settings in this VM will be completely outdated. Thanks.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Adrian, I was wondering earlier today already and now it's clear. The templates you have used are not the ones which I have updated recently. At least the 12.04 32bit one does not include the changes made by bug 911795.

Can you please check that?
Status: REOPENED → ASSIGNED
Flags: needinfo?(afernandez)
> mm-ub-1304-64-4.qa.scl3.mozilla.com

That one is online now.

> mm-ub-1204-32-4.qa.scl3.mozilla.com

Michael, I have huge issues in getting Firefox to connect to our http://mm-ci-master.qa.scl3.mozilla.com:8080 website. Firefox is always trying to use the proxy for that which will fail with Access denied. I have absolutely no idea why Firefox is not taking the no_proxy or ignore_list from the system. Do you have any idea?
Flags: needinfo?(mhenry)
(Assignee)

Comment 11

4 years ago
Henrik, do you want me to convert the 12.04 32 bit template back to a VM?
I am checking the VM task/events log and the VMs were indeed all converted back to templates on September 5th (as per the last bug comment in Bug 911795). Perhaps your changes were missed on this one?

As for Firefox not working with the proxy, did you re-apply the changes that were previously supposed to be made and see if it works now?
Flags: needinfo?(afernandez)
(Assignee)

Comment 12

4 years ago
Forgot to add, 

mm-win-7-64-4.qa.scl3.mozilla.com will be re-deployed.
mm-ub-1204-64-4.qa.scl3.mozilla.com was at login prompt, I logged in and rebooted the node and VNC is working.

If there's anything else that needs to be checked (minus the proxy), please let me know.
(Assignee)

Comment 13

4 years ago
mm-win-7-64-4.qa.scl3.mozilla.com back online.

Leaving bug open until proxy issue could be resolved but if that's indeed an issue, it should be a separate bug.
(In reply to Adrian Fernandez [:Aj] from comment #11)
> Henrik, do you want me to convert the 12.04 32 bit template back to a VM?

I think that would be good. So I could check what's wrong there.

> I am checking the VM task/events log and the VMs were indeed all converted
> back to templates on September 5th (as per the last bug comment in Bug
> 911795). Perhaps your changes were missed on this one?

Those were 4 VMs and I clearly did everything sequentially to not miss something. So I really don't understand this. :(

> As for Firefox not working with the proxy, did you re-apply the changes that
> were previously supposed to be made and see if it works now?

All the proxy settings are totally equal to the other boxes, but Firefox is still not able to reach our jenkins website.

I will work on the integration of the other VMs now.
(Assignee)

Comment 15

4 years ago
As per yesterday, issue is not with the proxy but a local issue.
mm-ub-1204-32-4.qa.scl3.mozilla.com
mm-ub-1304-64-4.qa.scl3.mozilla.com

Can connect now. I know we did some troubleshooting but my work-week got the best of me.
Did you determine culprit?
Flags: needinfo?(mhenry)
Actually not, given that I was waiting for your reply. :) You are most likely not around now, so I will do further checks regarding the /etc/resolve.conf differences we have seen.
Looks like the command which we missed to execute is:

gsettings set org.gnome.system.proxy ignore-hosts "['localhost','127.0.0.0/8','10.0.0.0/8','*.scl3.mozilla.com','*.phx1.mozilla.com']"

I also updated the tail file for resolve.conf.

Adrian, can you please check that also for the templates? It should be wrong there too. Also it would be great when we get our Mana documentation updated, so it specifies what we really have to do. As of now it looks kinda nontransparent. 

I would suggest that we do all the proxy config settings via the terminal. As best via a script, which we would need for Puppet anyway, right?
Flags: needinfo?(afernandez)
(Assignee)

Comment 18

4 years ago
Following VM templates updated;
Ubuntu_12.04_x86
Ubuntu_13.04_x86-64

As for the documentation, it does list; gsettings set org.gnome.system.proxy ignore-hosts etc
so perhaps an update (apt-get etc) blew the changes.

As for what needs to be done, pretty much everything that's listed in the proxy side of things need to be done on any NEW template we are creating.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago4 years ago
Flags: needinfo?(afernandez)
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.