Closed Bug 726692 Opened 12 years ago Closed 12 years ago

move cb-aussstage01 out of sjc1

Categories

(Infrastructure & Operations :: Virtualization, task)

Product:
Infrastructure & Operations
Component:
Virtualization
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: afernandez)

References

Details

cb-ausstage01 is in sjc1, and will need to move.

This is hit by seamonkey and a few contributed builds of Firefox, via aus2-community.mozilla.org.

We need a plan to move this.
apologies for the cc spam based on who has logins there.  Please add other affected parties.
Just to be clear, it is also ssh-hidden behind the community jumphost at present.
<Callek> dustin: fwiw with the volume we have for SeaMonkey aus pings, we can live with a few hour downtime at practically any time of day
<Callek> dustin: reluctantly ok with even a 24 hour or so downtime
<dustin> ok
<Callek> the key though is to please try hard not to make within a week right after an uplift day
<Callek> its *ok* if it has to happen, but I'd rather have a week of leeway where users are not shocked that updates are not offered for a major update
(In reply to Dustin J. Mitchell [:dustin] from comment #1)
> apologies for the cc spam based on who has logins there

If this is the update service for the toolkit update component, I'm surprised that Sam and I have logins to it; I certainly didn't know I did!  

Camino doesn't use toolkit update component or corresponding update web infrastructure, so unless the logins are part of some "all the services that the community users need, for all community users" LDAP group, Sam and I probably don't need to have logins for it ;-)
This is a VM with a mess 'o disk.  The best option will be to just move it as-is.

This is on the 'storage1' datastore, which is local to cb-vmware01, so moving may be tricky.  The host isn't puppetized, and has been built over a long time, so rebuilding's not a great option, either.  Dan, what do you think is the best path here?
I can use VMware Converter to migrate it to the new vSphere cluster in scl3. It will take about 1 to 4 hours, during which time all services on the source host need to be off.
Summary: move community AUS out of sjc1 → move cb-aussstage01 out of sjc1
Cool.  For the record, this is ~40G - I'm not sure what I meant by "a mess" :)

Let's plan to do this mid-next-week.

Callek, I know SM is on this host.  Smokey indicates camino's not.  Do you happen to know who else we need to announce a downtime to?

Also, what URL is SM using for AUS checks?  We'll need to change that to point to the new host.
ah, I knew the URL already:
  aus2-community.mozilla.org
so we'll need to adjust that CNAME to point to ausstage1.community.scl3.mozilla.com.
(In reply to Dustin J. Mitchell [:dustin] from comment #8)
> ah, I knew the URL already:
>   aus2-community.mozilla.org
> so we'll need to adjust that CNAME to point to
> ausstage1.community.scl3.mozilla.com.

Yes adjusting the CNAME should be enough. Besides SeaMonkey only old Calendar standalone builds, and some (relatively old) Firefox/Thunderbird contrib/community-ports are on here. The numbers of non-seamonkey are <1% of total traffic to this host (last I checked) so I think a few hour to a day downtime does not need any other correlation.

Besides, everyone who has access is CC'ed to this bug.
flows that I know of for the new host are in bug 742571
Virtualization Gurus -- the flows are sufficiently in-place for this change to occur.  Can you pick (and announce here) a time to move this VM to scl3 and rename it?  I'll get inventory updated with the new info.
Assignee: dustin → server-ops
Component: Server Operations → Server Operations: Virtualization
QA Contact: cshields → dparsons
destination for this VM is
 ausstage1.community.scl3
 63.245.223.9
When can we do this?
I'm so good, I answered that before you asked it!

(In reply to Dustin J. Mitchell [:dustin] from comment #11)
> Virtualization Gurus -- the flows are sufficiently in-place for this change
> to occur.  Can you pick (and announce here) a time to move this VM to scl3
> and rename it?  I'll get inventory updated with the new info.
Phong, can we get this scheduled?
Let's do this on Thursday starting at 9am PDT.

If the migration looks like it will extend overnight, we'll need to restart the old (sjc1) VM while continuing to try to migrate to scl3.

Phong, is that time OK?  Who will do this?
Assignee: server-ops → mburns
This is planned for just over an hour for now.

mburns: I don't believe there's any advance shutdown work required, so you should be good to go.
mburns was running into flow issues, taking this over.
VM will go down shortly.
Assignee: mburns → afernandez
VM down and migration started.

cb-aussstage01 -> ausstage1.community.scl3.mozilla.com


Status: 2% Complete
ETA: 1 hr (will fluctuate)
Migration completed. 

Getting VM online, CNAME was updated as well.
VM is online.

Please verify that all is well.

If no reply is received, bug will be closed in 24 hours.
dustin@cerf ~ $ nc -vz aus2-community.mozilla.org 80
Connection to aus2-community.mozilla.org 80 port [tcp/http] succeeded!
dustin@cerf ~ $ nc -vz aus2-community.mozilla.org 443
Connection to aus2-community.mozilla.org 443 port [tcp/https] succeeded!

and I can SSH from the jumphost, too.  I'll go one further and call this done.  Anyone using this host for AUS, let me know if you see problems - either in IRC or a new bug.

Thanks, Aj!
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
(In reply to Dustin J. Mitchell [:dustin] from comment #22)
> dustin@cerf ~ $ nc -vz aus2-community.mozilla.org 80
> Connection to aus2-community.mozilla.org 80 port [tcp/http] succeeded!
> dustin@cerf ~ $ nc -vz aus2-community.mozilla.org 443
> Connection to aus2-community.mozilla.org 443 port [tcp/https] succeeded!
> 
> and I can SSH from the jumphost, too.  I'll go one further and call this
> done.  Anyone using this host for AUS, let me know if you see problems -
> either in IRC or a new bug.

I am seeing ssh problems from my .nl hosts, I'm filing a new bug for that
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.