Closed Bug 883103 Opened 11 years ago Closed 11 years ago

Upgrade templates for Ubuntu 12.10 x86/x86_64 to 13.04 in qa.scl3.mozilla.com

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: whimboo, Assigned: afernandez)

References

Details

(Whiteboard: [qa-automation-wanted]) 

Ubuntu 13.04 has been released for a while and we are running tests successfully with that new version for a while now in staging. We want to upgrade our production machines to 13.04 too.

So can we please get the Ubuntu 12.10 VM templates updated to 13.04? Thanks.
Summary: Upgrade templates for Ubuntu 12.10 x86/x86_64 to 13.04 in qa.scl3,mozilla.com → Upgrade templates for Ubuntu 12.10 x86/x86_64 to 13.04 in qa.scl3.mozilla.com
This probably belongs in the SRE queue.
Assignee: server-ops-dcops → server-ops
Component: Server Operations: DCOps → Server Operations
QA Contact: dmoore → shyam
(In reply to Van Le [:van] from comment #1)
> This probably belongs in the SRE queue.
We do handle alot of VM work but it all goes through Dan and Greg first.
Over to VM team
Assignee: server-ops → server-ops-virtualization
Component: Server Operations → Server Operations: Virtualization
QA Contact: shyam → dparsons
:whimboo, are you asking merely for our Ubuntu template to be updated, which would effect new VMs, or are you asking for all your existing VMs to be updated?
(In reply to Dan Parsons [:lerxst] from comment #3)
> :whimboo, are you asking merely for our Ubuntu template to be updated, which
> would effect new VMs, or are you asking for all your existing VMs to be
> updated?

I'm not sure what the preferred way is here. Given that the template has to stay in sync we have to update the template. The VMs I can update myself, but I wonder if it would make more sense to distribute those from the template again.

As for now those VM's are named like mm-ub-1204-64-2 which would not match anymore after the upgrade. I wonder what's best for IT, to update the DNS name whenever a host gets an OS upgrade, or if we should step back and simply enumerate all the boxes.
OK, let's just update the template for now.

Moving to Server Ops so this can be completed
Assignee: server-ops-virtualization → server-ops
Component: Server Operations: Virtualization → Server Operations
QA Contact: dparsons → shyam
:whimboo just to clarify the work, do you want clean 13.04 installs or shall we do the following;
- clone the existing 12.10 template into a 13.05 && upgrade new template to 13.05 via apt-get

Also, what naming scheme would you like to us?

* We will still keep the 12.10 template around

Please advise, thanks!
Flags: needinfo?(hskupin)
(In reply to Adrian Fernandez [:Aj] from comment #6)
> :whimboo just to clarify the work, do you want clean 13.04 installs or shall
> we do the following;
> - clone the existing 12.10 template into a 13.05 && upgrade new template to
> 13.05 via apt-get

I think we can do the upgrade of 12.10. But please name those 13.04 and not 13.05. :)

> Also, what naming scheme would you like to us?
> 
> * We will still keep the 12.10 template around

Nope, once we have 13.04 we don't need 12.10 anymore. We will only keep 12.04 which is the LTS.

Thanks
Flags: needinfo?(hskupin)
This actually raises the question about our support agreement with our users on Linux. I would argue that we need to support the distributions where a large share of our users still exist. Before we throw away Ubuntu 12.10 can I talk to Metrics and perhaps Release Management to try to answer these questions?
When an OS manufacturer stops the support for a given release, we were also going to drop our support. That happened for OS X 10.5 and Windows 2000 some months ago. With the 13.04 Ubuntu release Canonical officially dropped support for 12.10 as of the docs. 12.10 is not a LTS release.
We can't weight too strongly "manufacturer support". Windows XP for example has ~25% of our user base even though Microsoft is dropping support. The criteria has to be whether the cost of support outweighs the size of the abandoned user population. I don't want to bikeshed on this conversation in this bug though.

Is it too much work for you to support all three versions at this time? If so, go ahead and just upgrade the 12.10 VMs. I think it's fine to support the latest Release and LTS versions in automation until otherwise requested.
It's most likely not a problem to have those VM running and executing tests, but as long we don't have puppet to keep the machines up to date, I have to update all of them manually. So here we talk about 6 additional VMs (3x x86, 3x x86_64). So I really would like to drop 12.10 in favor of 13.04.
For the record, Canonical plans to support Ubuntu 12.10 until April of next year[1]. Given this information I would assume we'll need to support 12.10 at least for another 9 months. 

I have a Metrics bug open to look into our user population on this version but I think we should proceed as if we need to support all three versions for the time being.

1. https://wiki.ubuntu.com/Releases
(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #12)
> For the record, Canonical plans to support Ubuntu 12.10 until April of next
> year[1]. Given this information I would assume we'll need to support 12.10
> at least for another 9 months. 

No, this has to be a failure in the table. As noted at the top of the page: "Normal Ubuntu releases are supported for 9 months". Given the release of 12.10 in October last year, the support ends this months. So we have a bit of overlap between normal releases, which I haven't known.
I don't think there is an error in the table. The move to a 9 month support cycle for interim releases went into effect as of Ubuntu 13.04 I believe. So Ubuntu 12.10 is the last release supported for 18 months (October 2012 - April 2014) and Ubuntu 13.04 is the first release supported for 9 months (April 2013 - January 2014).

From the Ubuntu Technical Board meeting notes for March 18, 2013:
> "The first vote was very crisp: Reduce maintenance period for 
> regular/standard (non-LTS) Ubuntu releases from 18 months to 9 months 
> (starting with release TBD)"

This vote passed.

> "The second vote was about the “when” and the TB was asked to vote on 
> Implementation of the above change to the maintenance schedule effective 
> in 13.04 release and later."

This vote also passed.

Source:
> http://fridge.ubuntu.com/2013/03/19/ubuntu-technical-board-looks-at-shuttleworths-proposal-for-release-management-methodology/
Thank you for those details Anthony. It was not clear that the support duration has been changed lately. So given that 12.10 is still supported until April 2014 we could keep the VMs for 12.10 if you really need those.

But keep in mind that Fedora VMs are also coming up, so we will get dozen of machines, which I don't think we can handle that good in our small team. At least not until we have Puppet ready.
Yeah, I think that's a fair point. I don't want you to be overloaded with maintenance. Until puppet is stood up how many Linux VMs could you reasonably support? We should balance resources against where the majority of our users exist.
For Linux the most recent version and the last LTS. That would apply to all distributions.

Adrian, have you already started? If not, then sorry for the noise here. But please get started with the upgrade of the templates. Lets get 12.10 replaced by 13.04. Thanks!
(In reply to Henrik Skupin (:whimboo) from comment #17)
> For Linux the most recent version and the last LTS. That would apply to all
> distributions.

I think that's perfectly reasonable until we have technology and tools in place to support scale. Thanks and sorry for the noise.
Assignee: server-ops → afernandez
13.04 templates ready for testing (upgrade from 12.10);
Ubuntu_13.04_x86    10.22.73.238
Ubuntu_13.04_x86-64 10.22.73.222

Please verify all is well so that we could convert to template and deploy from it.
I will verify those templates early next week. I'm on a mobile connection today so cannot do such heavy traffic tasks.
Adrian, those VMs are ready now and can be converted to templates. What would be the best way to replace the 12.10 ones? If we want to replace the existing ones we will have to do it step by step. Otherwise we could have three new VMs, which could be used for initial checks, and which then would replace the 12.10 VMs. I would kinda like to not hot-swap the VMs.
Creating new ones is not a problem. Will the following naming convention work?
mm-ub-1304-32-1.qa.scl3.mozilla.com
mm-ub-1304-32-2.qa.scl3.mozilla.com
mm-ub-1304-32-3.qa.scl3.mozilla.com
mm-ub-1304-64-1.qa.scl3.mozilla.com
mm-ub-1304-64-2.qa.scl3.mozilla.com
mm-ub-1304-64-3.qa.scl3.mozilla.com
Yes, that sounds perfect.
hosts are online with the correct hostname (needed to remove /etc/hostname from the templates in order for them to pick up the correct hostname from DHCP). Hosts added to inventory and https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service updated as well.

All should be well (ie all online), feel free to re-open if something is wrong. Open up a new bug if/when you need the other VMs to be decommissioned.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
(In reply to Adrian Fernandez [:Aj] from comment #24)
> hosts are online with the correct hostname (needed to remove /etc/hostname
> from the templates in order for them to pick up the correct hostname from
> DHCP).

Oh, so that means all other templates would also have to be updated regarding this change given that I had to update a couple of hostnames. Would you have time to check the 12.04 ones? We wouldn't have to do that for 12.10 given we will get rid of them soon.

> All should be well (ie all online), feel free to re-open if something is
> wrong. Open up a new bug if/when you need the other VMs to be decommissioned.

Thanks a lot! Looks good so far. I will file the new bug once the 13.04 VMs have been proven to be stable enough.
Status: RESOLVED → VERIFIED
Flags: needinfo?(afernandez)
The templates have the /etc/hostname file removed (with the previous updates that were done).
Flags: needinfo?(afernandez)
Thank you Adrian!
Blocks: 910277
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.