Closed Bug 1255114 Opened 9 years ago Closed 9 years ago

Provide new OSX 10.9 (Darwin13.x 64bit) system for NSS - (and delete nss-vm-darwin9-1.community.scl3)

Categories

(Infrastructure & Operations :: MOC: Projects, task)

Product:
Infrastructure & Operations
Component:
MOC: Projects
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: KaiE, Assigned: vinh)

References

Details

NSS uses a OSX VM (Parallels) that Mozilla has provided. The hostname is nss-vm-darwin9-1.community.scl3.mozilla.com It's a 32 bit VM running OSX 10.5, Darwin 9 Because OSX 10.5 is no longer supported by Mozilla, and because we are adding a feature to NSS that doesn't work on OSX 10.5, and because we don't have any other 32-bit OSX VM currently, I'd like to ask: Could you please upgrade the operating System on nss-vm-darwin9-1.community.scl3.mozilla.com to OSX 10.6, 32 bit? If upgrading the Parallels VM from OSX 10.5 to OSX 10.6 is possible, I think that will be best. Please note that you'll have to think about hostnames to avoid confusion. Because, we already have ANOTHER VM that's named nss-vm-darwin10-1 ... So, if you are required to change the hostname to include the new operating system version, it probably should be nss-vm-darwin10-2.community.scl3.mozilla.com
Summary: Upgrading nss-vm-darwin9-1.community.scl3.mozilla.com to Darwin 10 32bit → Upgrading nss-vm-darwin9-1.community.scl3.mozilla.com to OSX 10.6, Darwin 10 32bit
Blocks: 1255117
Sadly, I've wandered *very* far afield from being able to help with this particular request. I'm also not sure you can still find 10.6 install media, although I remember a conversation with gcox recently about a dmg of that name we found lying around on some random server. Anyway, it's worth reiterating for those who might work on it: NSS is a critical component of Firefox and we need to take this request seriously.
I squirreled a copy away in ESXland (note to future me: installmedia/deprecated/emergency_archive). :cknowles or I can make this available to whomever picks this up.
Assignee: nobody → ludovic
(In reply to Greg Cox [:gcox] from comment #2) > I squirreled a copy away in ESXland (note to future me: > installmedia/deprecated/emergency_archive). > :cknowles or I can make this available to whomever picks this up. I've picked it up. Let me know how to access it while I sort out how to remote to the machines (eg see if https://mana.mozilla.org/wiki/display/SYSADMIN/Connecting+to+Parallels is still accurate)
First thing I did, was make a snapshot. Then I loaded the iso (the iso image is now copied on //2 in /Users/administrator/) in // and clicked the install OSX I was asked to accept Apple legales ,and then to choose the disk to install the beast on. I chose the only available disk - whihc contains a Server version of OSX on which 10.6 from the CD refuses to install. Kaie, Dustin how are you advising me to proceed ?
Flags: needinfo?(kaie)
Flags: needinfo?(dustin)
I don't know why Dustin had used a Server version of OSX. Could it be a requirement of Parallels? If it's easier to setup a fresh VM, that works, too. I've backed up the most important stuff from the existing VM. Feel free to temporarily shutdown the existing VM nss-vm-darwin9-1.community.scl3.mozilla.com, to free up more RAM resources on the physical host, if necessary.
Flags: needinfo?(kaie)
found a server image battling with vnc and keyboard.
Mounted the ISO image, launched the process , the VM restarts and ask for login, it's like it's not booting the CD/DVD :( I'll continue investigating these tomorrow.
Grr can't get the vm to boot directly on the dmg, looking on //'s support site
I vaguely recall that you could only boot from an OS X install CD by burning the cd, putting it in the drive, and linking the drive to the VM. Parallels is not what one calls "quality" software.
Flags: needinfo?(dustin)
(In reply to Dustin J. Mitchell [:dustin] from comment #9) > I vaguely recall that you could only boot from an OS X install CD by burning > the cd, putting it in the drive, and linking the drive to the VM. Parallels > is not what one calls "quality" software. Grrrr ... ok.
Can someone from DCops burn the Server .dmg on //2 and insert burnt CD in the drive of the xserve and give me the bug back once done ?
Component: MOC: Service Requests → DCOps
QA Contact: lypulong
(and since it took me a few minutes to figure out, by "//2" ludo means "parallels2.community.scl3.mozilla.com" :))
:usul, i've inserted a dvd burner and a blank dvd. however i don't have the admin password to start the burning process. can you give me the password or remote desktop and finish the process? also, which server is the xserve? >insert burnt CD in the drive of the xserve and give me the bug back once done ?
(In reply to Van Le [:van] from comment #13) > :usul, i've inserted a dvd burner and a blank dvd. however i don't have the > admin password to start the burning process. can you give me the password or > remote desktop and finish the process? also, which server is the xserve? My bad it's https://inventory.mozilla.org/en-US/systems/show/6274/ > >insert burnt CD in the drive of the xserve and give me the bug back once done ? It's in community , I'll send you teh password via encrypted email.
mail sent
Had you had time to look at this ?
Flags: needinfo?(vle)
the image is too large for the 4.7gb dvds we have. i'll need to purchase some unless we can copy/dd this image to a USB stick?
eus might have some, i'll check mtv2 later today.
Flags: needinfo?(vle)
:usul, i was able to grab an 8gb dvd from desktop and successfully burnt the DVD.
:usul, i was able to grab an 8gb dvd from desktop and successfully burnt the .dmg.
(In reply to Van Le [:van] from comment #20) > :usul, i was able to grab an 8gb dvd from desktop and successfully burnt the > .dmg. Woot !!!
:( So I loged onto the vm started the install process and the vm when into shutdown reboot mode , but never really shat down. I tried to resytart shutdown it from the Parallels GUI, with no luck. I then rebooted the host. Host is now probably prompting soemthing to do with the DVD in the drive as it's unreachable. Sal, Vinh, do you guys still have access to scl3 , if so could one of you go there and do the update from the machine itself ? if Not I'll have to annoy van again :(
Flags: needinfo?(vhua)
Flags: needinfo?(sespinoza)
:usul, they don't really come down to the DC anymore. how can i help?
Make paralles2 ask for a login prompt again because right now the boot process of the physical machine is probably waiting for human interaction.
Flags: needinfo?(vhua)
Flags: needinfo?(sespinoza)
should be back up. i found the mini powered off.
I am miserably failing at making the vm start from the CD/DVD and thus launch the update. I can't find any relevant docs that tells me what to fix to have this work. Suggestions welcomed.
Remind me, is this the only VM on the host? Perhaps we could just install 10.6 on the host itself? Or is that hardware too new to run 10.6? /me shakes fist impotently at Big Fruit
(In reply to Dustin J. Mitchell [:dustin] from comment #27) > Remind me, is this the only VM on the host? Perhaps we could just install > 10.6 on the host itself? Or is that hardware too new to run 10.6? Yeah it's the only vm. No idea if the hardware can run 10.6 or not, do you have a pointer on how to check that ?
Does http://www.everymac.com/mac-answers/snow-leopard-mac-os-x-faq/mac-os-x-snow-leopard-fully-compatible-intel-macs.html help? Amy may also have some ideas. Or, maybe just give it a try? I expect the DVD just won't boot if it doesn't work. Sorry this is so frustrating. By the way, I just checked with MacInCloud and they seem to support only the latest OS X release.
Flags: needinfo?(arich)
Host is running 10.7 so we should be good. I just don't see how I'll be able to do the install on the physical host via vnc ...
Also Kaie wants 32 bits so I need to e careful, not to get a 64 bit env installed.
I don't think I have any additional info here, sorry. We don't use virtualization, so it's been a very long time since I've touched it (and then I was using fusion). If it's the only vm, you might be much better off just installing it on the bare metal.
Flags: needinfo?(arich)
I was looking for information about compatibility -- if 10.6 won't run on this hardware, there's no point trying.
Component: DCOps → MOC: Projects
QA Contact: lypulong
As far as I can tell there's no way to do a 32bit only 10.6 install. You can boot into a 32bit kernel: https://support.apple.com/en-gb/HT3773 I don't see the exact model of the hardware in inventory (there are many versions of mac mini) to be able to say if it will run 10.6. The hardware was bought in 2012 if inventory is correct and 10.6 was released in 2011 so it seems likely. Probably the simplest way to move on with that would be to ask dcops to put one of the IP accessible graphics console servers onto it (not sure what they call those) and access the hardware remotely.
Apple doesn't support backwards hardware compatibility with their releases, so if the hardware is NEWER than the OS, there's a good chance that it won't run the OS (if there was a new version of the OS released already). You can tell what kind of hardware it is by running system_profiler on the box (look for "Model Identifier:" ).
(In reply to Amy Rich [:arr] [:arich] from comment #35) > Apple doesn't support backwards hardware compatibility with their releases, > so if the hardware is NEWER than the OS, there's a good chance that it won't > run the OS (if there was a new version of the OS released already). Yeah, but sometimes when it is close enough it works and it just says the hardware was bought in 2012 not that it was a 2012 model. > You can tell what kind of hardware it is by running system_profiler on the box (look > for "Model Identifier:" ). Not logged into the machine. Usul?
Macmini5,3
https://en.wikipedia.org/wiki/Mac_Mini Shipped with 10.7. Last one to officially support 10.6 was the MacMini4,1 though there are hacks to get around that: http://www.fatheaddesign.com/659/installing-os-10-6-8-on-a-mac-mini-built-for-10-7-yes-its-possible/ but that's irritating and difficult to support at best. Perhaps we could replace Parallels with a better virtualisation system (vmware)? I believe we can't just run a VM on our existing vmware infrastructure, legally, since Apple only allow it on Apple hardware.
I appreciate your initiative, but it sounds like this request is very difficult to accomplish (which I didn't expect). Maybe we should switch to a backup plan. I have a Mac mini intel 32bit at home, which is capable of running OSX 10.6 bit. Maybe we should use that one and I'd run the buildbot slave from my home network. In case anything goes wrong with my hardware during the next year (until end of support of Firefox 45 in spring 2017), we should probably keep the darwin 10 machine (and wontfix bug 1255117) And we could delete this darwin9 VM, and reinstall it with a parallels VM running OSX 10.9 64bit. Would that be possible? To summarize, the updated plan would be: - OSX 10.6 32bit - Kai's Mac mini at home - OSX 10.6 64bit - keep nss-vm-darwin10-1.community.scl3.mozilla.com - mark bug 1255117 wontfix - morph this bug 1255114 to: - delete nss-vm-darwin9-1 - replace with a fresh install of OSX 10.9.x, Darwin 13
The more general issues here are trying to run server stuff on a mac mini isn't going to go well for anyone and using parallels is always going to be painful. If there is any of that we can fix then it's better to do so and reduce the support complexity.
If I understand correctly, we currently have two OSX parallels hosts, where each of them is running exactly one OSX VM? (nss-vm-darwin9-1 and nss-vm-darwin10-1). Is this correct? Then we should ask Justin, why he had chosen that architecture in the first place, instead of allowing us to use the hardware directly. Was it done, because it was anticipated we might request additional VMs? Or was it done for isolation purposes?
I assume by "Justin" you mean me (there are at least two Justin's working here, but they're not involved)? Yes, at one point we had more VMs, and also wanted the redundancy of being able to move VMs between hosts if that became necessary due to hardware failure. Parallels has its issues, but when it works it does allow us to manipulate the guests without hands onsite -- for example, the guest restarts that have been requested every now and then. If I recall, compatibility is better under parallels, too -- as in this case where, you can run 10.6 in a VM, but not on the bare hardware. In releng, we run all of our macs on bare hardware, but we also have 100's of them and we overspend to ensure we'll have enough of a particular model to last until we expect to no longer support that version of the OS. We always under-estimate, and we have "a few" failed machines at any given time and frequent datacenter trips. Scale makes the problems of running macs in production a little bit more manageable day-to-day, but it's still a nightmare. Hopefully that's helpful context?
(In reply to Kai Engert (:kaie) from comment #39) > I appreciate your initiative, but it sounds like this request is very > difficult to accomplish (which I didn't expect). > > Maybe we should switch to a backup plan. > > I have a Mac mini intel 32bit at home, which is capable of running OSX 10.6 > bit. > Maybe we should use that one and I'd run the buildbot slave from my home > network. > > In case anything goes wrong with my hardware during the next year (until end > of support of Firefox 45 in spring 2017), we should probably keep the darwin > 10 machine (and wontfix bug 1255117) > > And we could delete this darwin9 VM, and reinstall it with a parallels VM > running OSX 10.9 64bit. Would that be possible? > > To summarize, the updated plan would be: > > - OSX 10.6 32bit - Kai's Mac mini at home > > - OSX 10.6 64bit - keep nss-vm-darwin10-1.community.scl3.mozilla.com > > - mark bug 1255117 wontfix > > - morph this bug 1255114 to: > - delete nss-vm-darwin9-1 > - replace with a fresh install of OSX 10.9.x, Darwin 13 I'm fine with that. And that's where we are going to go with.
Justin, thanks a lot for comments, very helpful context!
I'm morphing this bug according to the udpated plan: Please delete nss-vm-darwin9-1.community.scl3. Please install a new OSX 10.9 Darwin13.x 64bit VM for NSS It's up to you to decide if you prefer to install it as a "parallels VM" or installed it on hardware directly. (According to Justing, installing as a parallels VM would allow us to restart a stuck system more easily.) Thanks in advance.
Summary: Upgrading nss-vm-darwin9-1.community.scl3.mozilla.com to OSX 10.6, Darwin 10 32bit → Provide new OSX 10.9 Darwin13.x 64bit VM for NSS - (and delete nss-vm-darwin9-1.community.scl3)
Summary: Provide new OSX 10.9 Darwin13.x 64bit VM for NSS - (and delete nss-vm-darwin9-1.community.scl3) → Provide new OSX 10.9 (Darwin13.x 64bit) system for NSS - (and delete nss-vm-darwin9-1.community.scl3)
So...the mac mini that is hosting nss-vm-darwin9-1.community.scl3 (https://inventory.mozilla.org/en-US/systems/show/6274/) is no longer booting into OS. It's flashing the "? folder", indicating there's no OS to boot from. I'm trying to search around for a 10.9 dmg to install onto the physical mini. Can I get a green light to proceed?
Flags: needinfo?(kaie)
I don't know if you need green light from anyone else, but green light from me. Thanks.
Flags: needinfo?(kaie)
Last known status from vinh was that the machine didn't boot after a reboot because it wasn't finding a kernel to boot from. I guess vinh was then looking for some install media to make a proper install of 10.9. He's now off for the week-end so no news for this bug until at least next monday.
Ok ran into some hiccups but finally got OSX 10.9 installed onto the mac mini. 10.9 is installed onto bare hardware, no VM. Hostname is "parallels2.community.scl3.mozilla.com". Right now there's a generic account (Ludo ping me for the user/pw). FYI: When trying to install OSX 10.9, I ran into this error "This copy of the Install OS X Mavericks application can't be verified. It may have been corrupted or tampered with during downloading". The fix is to go into terminal (at install screen) and change the date to when the installer was created. For the installer that I obtained, the date was aug 27, 2014 8:57am. "$date 082720572016" <--format:[date [month][day][time][year]
Flags: needinfo?(ludovic)
Side note: users have reported that they had to change the date in terminal to match ntp dates for the fix but that did not work for me. Had to follow this user's tip instead: This worked for me when I realised I should check the date on the copy of "Install Mavericks.app" I had downloaded some months ago. I used the "Date Modified" including the time, input this information into the installer's Terminal window - in my case the file was downloaded on December 31st 2015 at 20:14 - so I entered "date 123120142015", and all was fine from there on in. Adding today's date/time or the date/time given above did NOT work. If you're having trouble with these - check the date/time on the installer you're using.
It's like the old copy-protection schemes on floppy-disks. Put me down for "I hate you, Apple".
Vinh the password needs to go into the community gpg file once there let me know and I'll send kaie an ecrypted email containing the password.
Flags: needinfo?(ludovic)
creds sent to kaie.
(In reply to Ludovic Hirlimann [:Usul] from comment #53) > creds sent to kaie. Couldn't decrypt, wrong key used. Could you please resend using gpg key 0x1C27423725007724 ? Thanks
Thanks a lot, password received, I'm able to login to the machine using ssh. It might be necessary to log in to the machine and install some software, in particular, I'll need "macports" installed. For the other darwin10 I was given vnc access. Would it be possible to enable vnc access on this new darwin13 machine? I sent an encrypted email to Ludovic that contains the vnc password that we're already using on the old darwin10 machine. Thanks
Also, could you please install the development tools that come on the OSX CD, I think it's called Xcode? Thanks
Assignee: ludovic → vhua
:kaie - VNC is enabled and I was able to remote in (user/pw is the same one Ludo sent you). As for Xcode, are you ok with the current version that's provided in App Store?
Flags: needinfo?(kaie)
(In reply to Vinh Hua [:vinh] from comment #57) > :kaie - VNC is enabled and I was able to remote in (user/pw is the same one > Ludo sent you). I've just tried to connect to parallels2.community.scl3.mozilla.com port 5900 using Linux command "vncviewer" (which I've used in the past and which still works with darwin10). It fails with this error output: TigerVNC Viewer 64-bit v1.6.0 Built on: 2016-01-04 15:10 Copyright (C) 1999-2015 TigerVNC Team and many others (see README.txt) See http://www.tigervnc.org for information on TigerVNC. Wed Apr 20 19:11:56 2016 CConn: connected to host 127.0.0.1 port 5904 Wed Apr 20 19:11:57 2016 CConnection: Server supports RFB protocol version 3.889 CConnection: Using RFB protocol version 3.8 CConnection: No matching security types CConn: No matching security types FYI, it works with the other darwin10 machine, where I get: Wed Apr 20 19:12:02 2016 CConnection: Choosing security type VncAuth(2) Are there any security settings on this new darwin13 machine that restrict the vnc protocol to something specific, which my vncviewer application might not support?
(In reply to Vinh Hua [:vinh] from comment #57) > As for Xcode, are you ok with the current version that's > provided in App Store? I'm not sure. The purpose of this machine is to test compatibility of NSS with the same (potentially old) system libraries that Firefox uses to build. Since 10.9 will soon be the oldest OSX version supported by Firefox, it would be good to use whatever version is the standard on OSX 10.9. I don't know if using the latest XCode from the app store will still ensure that we build binaries that are compatible with OSX 10.9. Should we try to involve a Mac developer who might know the answer? On the other hand, maybe the app store software on OSX is sufficiently smart, and will install a version on this system that ensures compatibility with OSX 10.9?
Flags: needinfo?(kaie)
:kaie - can you ping me on irc (nick - vinh)?
Got VNC working for Kai. I needed to set a VNC password under "Sharing > Remote Management > Computer Settings > VNC viewers may control screen with password" Now onto getting Xcode installed.
Vinh, thanks a lot for your help getting Xcode. It's installed, and the new machine is working! It has been added to NSS CI, see osx-109 on this page https://bot.nss-crypto.org:8011/waterfall?reload=180 Marking as resolved fixed. Thanks to all of you!
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.