Closed Bug 885599 Opened 11 years ago Closed 11 years ago

Create VMs for Windows 8.1 preview (x86, x86-64) in qa.scl3.mozilla.com

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Windows 8
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: u279076, Assigned: afernandez)

References

(
URL
)

Details

(Whiteboard: [qa-automation-wanted]) 

I have been advised that Windows 8.1 will enter a Preview window on June 26 and release on August 1st, assuming things go according to plan. In anticipation for this I'd like us to stand up 1-3 nodes in Mozmill-CI (as many as we can support) as soon as possible after the Windows 8.1 preview becomes available. 

I think we will want to be able to test both Windows 8 and Windows 8.1 in Mozmill-CI/On-Demand for the short-term, at least until ADIs inform us it's no longer necessary to test both.

Thank you in advance.
Accordingly to our already existing machines in SCL3 which you can find listed in the Mana page, 6 more machines sound perfect. I say 6 because we would need 3 VMs for the 32bit version and another 3 VMs for the 64bit version.

Given that IT has to create those I will move this request forward.
Assignee: nobody → server-ops-virtualization
URL: https://mana.mozilla.org/wiki/display...
Component: Infrastructure → Server Operations: Virtualization
Product: Mozilla QA → mozilla.org
QA Contact: dparsons
Summary: Stand up some Windows 8.1 preview VMs → Create VMs for Windows 8.1 preview (x86, x86-64) in qa.scl3.mozilla.com
Version: unspecified → other
Whiteboard: [qa-automation-wanted]
Looks fine to parallel with the earlier ones*.  The usual caveats of being out on the bleeding edge of guest OS'es applies (i.e. "I hope this works").

*https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service
Assignee: server-ops-virtualization → server-ops
Component: Server Operations: Virtualization → Server Operations
QA Contact: dparsons → shyam
Once ISOs available, will template the preview release, QA verifies template is good and then deploy the requested VMs.
Assignee: server-ops → afernandez
Thank you Adrian!
Status: NEW → ASSIGNED
Following VMs created and currently being installed;
Win_8.1P_x86
Win_8.1P_x86-64

Once installation finishes, the proxy settings as per https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service#QAAutomationESXService-ProxySettings will be configured.

You will then have access to modify the VMs to your liking and then we shall convert to templates and deploy the 6 vms.
Templates currently online as follows;
Win_8.1P_x86 10.22.73.236
Win_8.1P_x86-64 10.22.73.231

All the proxy settings and power options are set as requested as per https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service

Please configure the rest is configured to your liking and verify that all is well.

Once all is confirmed good by you, we will them convert the VM into templates and deploy from there. If there's something wrong and/or need something else tweaked, please let us know.
Adrian, something doesn't work with the proxy settings. Even setting those again via the internet settings I cannot access local machines like fs1.qa.scl3.mozilla.com. Can you please check that? Thanks.
Henrik how were you trying to access the local machines?

I was able to do the following on both VMs;
Open Windows Explorer -> Select Network -> See all local VMs
"Start" -> search; \\fs1.qa.scl3.mozilla.com  | connect to said VM, see and access share.

No changes were done by me or by anyone else (that I am aware of, unless you changed something).

Please advise, thank you.
(In reply to Adrian Fernandez [:Aj] from comment #8)
> Open Windows Explorer -> Select Network -> See all local VMs
> "Start" -> search; \\fs1.qa.scl3.mozilla.com  | connect to said VM, see and
> access share.

Sure, but you cannot map this as a network drive to Z:\. Then you will get a failure. Might be a problem with the 8.1 preview.

For now I will not map it as a drive so I can continue installing our tools.
I installed all the necessary software on the 32bit machine, and tried to do some testruns. Sadly I'm not able to because the VM is crashing and rebooting all the time. Not sure what this is but the current behavior is far from being acceptable. With such instabilities we will not be able to run our tests. I'm not sure if there is anything we can do as of now. Adrian, does the ESX cluster officially support 8.1 yet? Could that be the problem?
Also some more things:

1. On both templates I miss audio devices

2. The proxy settings doesn't seem to work that well, given that mozdownload is not able to reach the ftp server to download firefox builds. Just change into c:\jenkins\workspace\mozilla-aurora_remote\mozmill-env\ and double click run.cmd. Then type 'mozdownload -v 21.0' which should successfully connect to the FTP server and download the 21.0 release for windows.

While testing all that I got another 10 or more crashes across both VMs.
8.1 being so new is technically not "officially" supported. Windows 8 is supposed which means that 8.1 being a .1 release shouldn't really break support but apparently it is.

As for the network drive mapping. I was able to map shared folder from fs1.qa.scl3.mozilla.com as the Z: drive and access the files via the mount.

As for the proxy not working, seems the system variables for the proxy did not stick. They will stick now, however, while (re)adding them, the VM crashed.

Will look into what's causing the crash and update.
For reference the "Blue" screen of death has the generic message with the following error;
CRITICAL_STRUCTURE_CORRUPTION

From the event log, got a more detailed error (well got the error number), landed on http://msdn.microsoft.com/en-us/library/windows/hardware/ff557228%28v=vs.85%29.aspx

Went with the hunch that the NIC driver (VMXNET 3) is to blame but after switching it to E1000E, after a few minutes, the Win_8.1P_x86 vm crashed (had the network mapped directory open).

However, the Win_8.1P_x86-64 (which previously crashed on me as well) as remained up. Both VMs now have the proxy environmental variables set but the seems the x64 one is looking for non existing win64-x86_64 directory in the ftp tree.

Please let me know if the x86-64 VM crashes for you.
Hm, I cannot connect to both of those machines. :(
Being that these are temporary, DHCP gave a new IP;
Win_8.1P_x86 10.22.73.214
Win_8.1P_x86-64 10.22.73.237

Since the testing may take a big longer than anticipated, I  have made these IPs static.
You should now be able to reach the nodes again.
Ok, I have access to the VMs and will run some tests to see how stable those are now. Something I have seen is that there are still no audio devices working. Haven't you had time to check that yet, or is it not supported as of now?
I also had to update the proxy settings of both templates to bypass local addresses.

Tests are running now. I will report back later.
Adrian, after some testing I haven't seen any crash anymore. So looks like we are stable enough now to get tests running. Thanks for the investigation and fix of the networking crashes.

If we can fix the audio issue that would be nice, otherwise we are good to go.
When you create the VMs can you name them like 

mm-win-81-32-1
mm-win-81-32-2
mm-win-81-32-3
mm-win-81-64-1
mm-win-81-64-2
mm-win-81-64-3
Being that the production Windows 8 machines don't have audio either, left the 8.1 preview VMs without it as well. We cannot easily add audio without hacking the vmx file.

Following are now templates (vmdks are aligned);
Win_8.1P_x86
Win_8.1P_x86-64

The requested VMs have been deployed from the above templates;
mm-win-81-32-1.qa.scl3.mozilla.com
mm-win-81-32-2.qa.scl3.mozilla.com
mm-win-81-32-3.qa.scl3.mozilla.com
mm-win-81-64-1.qa.scl3.mozilla.com
mm-win-81-64-2.qa.scl3.mozilla.com
mm-win-81-64-3.qa.scl3.mozilla.com

Confirmed up and remote desktop confirmed working.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Forgot to add;

VMs were added to inventory and https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service updated to reflect the new VM(s)+Template(s).
All 3 VMs which should run a 32bit version of Windows 8.1 are actually 64bit systems. That means we currently have 6 VMs with Windows 8.1 64bit.

I will wait with the final work until this problem has been solved.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
As per irc, looks like the template was also 64 bit (so both templates are 64 bit). Cannot even fathom how both templates ended up being 64 bit ... my apologies for this.

I have reinstalled "template" Win_8.1P_x86 with 32 bit Windows 8.1 Preview and verified it's indeed running 32 bit. (looks like 32 bit audio drivers got installed this time).

You could reach the template via temporary IP: 10.22.73.207

Please update any remaining needed settings, give the ok and we'll convert to a template and deploy from there.
Ok, should be all done now. So we can convert this back to a template and create the VMs. Regarding the audio device this seems to be a client issue with the VNC software. Looks like audio is not abled for me so I don't have that available. Means we don't have to worry about it.
Seems it doesn't have the modified "start" menu option that the previous template has. Shall we still proceed?
Good call. Missed that at the end. I installed an older version of the Classic Shell which seemed to be incompatible with Win 8.1. Now it works.
Sweet!

The 3 VMs have been redeployed from the updated template.

Please verify all is well and we'll close this out.
I cannot reach any of those machines. So I assume they got a different IP address assigned as documented on Mana.
Status: REOPENED → ASSIGNED
All connected 64bit machines have issues with the installed version of Java, which crashes and constantly disconnects the machines. Actually not sure what that is.

Dave, would you mind to check if this is caused by Jenkins and if we have to upgrade our version? Not sure if I find the time before I head into PTO. Thanks.
For comment 28, referring to the 32 bit machines? As I verified they were reachable via ping and rdp yesterday. I could still ping and rdp into them today;

--- mm-win-81-32-1.qa.scl3.mozilla.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 19.908/19.908/19.908/0.000 ms

--- mm-win-81-32-2.qa.scl3.mozilla.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 27.832/27.832/27.832/0.000 ms


--- mm-win-81-32-3.qa.scl3.mozilla.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 21.264/21.264/21.264/0.000 ms

As for the 64 bit machines, very possible that there might be unknown java issues with 8.1 Preview.

Are the actual VMs crashing or just java? As if just java, not much we could do on our end.
Oh sorry. Same problem again with the certificates! We have replaced the VMs so the stored certificates are invalid. Remmina didn't ask me again so I had to remove them manually from known_hosts. Now it works.

Regarding Java Dave and myself are working on it. If we find a solution we might have to update the template. Shall we leave the bug open until then or close it now?
Lets close for now and re-open if template(s) need to be updated.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
We got another Windows blue screen today at 6:06am PDT. See:

Information	7/26/2013 6:06:10 AM	Windows Error Reporting	1001	None

Fault bucket 0x109_7, type 0
Event Name: BlueScreen
Response: http://wer.microsoft.com/responses/resredir.aspx?sid=10&Bucket=0x109_7&State=1&ID=e7117520-92a9-4a1b-b482-a84a53101cb8
Cab Id: e7117520-92a9-4a1b-b482-a84a53101cb8

This is not related to the Java crashes reported earlier. Can you please investigate that? The other Win8.1 64bit VMs seem to run stable for now, except the Java issue.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Which VM got the bluescreen?
Oh, it was the 64-2 one.
It could have been a blip but we would need to access the server in order to check a few things (would need an OK in order to not disrupt possible tests that are running). Also keep in mind as before, that this is a preview and it could indeed be filled with bugs.

Has the same VM experience any issues or any other 8.1 VMs?
Feel free to connect to the appropriate machine. None of them actually are running tests as long as bug 900295 is not fixed.
Status: REOPENED → ASSIGNED
Just have seen that the hostnames on those nodes were not setup correctly and fixed those. But now we get system errors:

The name "MM-WIN-8-32-1  :0" could not be registered on the interface with IP address 10.22.73.62. The computer with the IP address 10.22.73.36 did not allow the name to be claimed by this computer.

Adrian, can you please check why those messages appear? I wonder if those are related to the crashes we have seen with the network interfaces.
Flags: needinfo?(afernandez)
As per the error, system with IP: 10.22.73.36 is mm-win-8-32-1.qa.scl3.mozilla.com which is named MM-WIN-8-32-1

System with IP: 10.22.73.62 is mm-win-81-32-1.qa.scl3.mozilla.com so it should be named: MM-WIN-81-32-1

So Windows' is just complaining that another system on the network is using the same name.
Flags: needinfo?(afernandez)
Adrian, I think we can close this bug as fixed now. Any remaining work will really happen on other bugs. Thanks again for your help!
Status: ASSIGNED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.