Closed Bug 1544789 Opened 7 years ago Closed 6 years ago

Please backup and decommission ship-it v1 and product-details

Categories

(Infrastructure & Operations :: Virtualization, task)

Product:
Infrastructure & Operations
Component:
Virtualization
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rail, Unassigned)

Details

(Whiteboard: [vm-delete:2])

Looks like we are ready to decomm ship-it v1! \o/

These are the things that come to my mind:

  1. shipit1.webapp.mdc1.mozilla.com: the web server
  2. the corresponding DB (looks like it's on the same machine)
  3. DNS entry for ship-it.mozilla.org - we switched to https://shipit.mozilla-releng.net/
  4. AFAIK, Ludo has already turned off the corresponding Nagios checks
  5. Champagne? :)

Thank you for giving home to this beast, that served us and saved lots of human time! :)

I'll kill the DNS whenver the webserver is off

Assignee: server-ops-webops → eziegenhorn

Backups run every night for this host, so whenever we shut it off, I'll disable the backups via the bacula console then.

Is there anything else we should do to close the bug? We haven't been using shipi-it v1 for more than a month now and had no issues :)

Yes I still have to clean up stuff, but it'll need to wait a bit longer due to some higher priorities.

No rush at all. I'm just curious.

Removed ship-it from puppet in ff558c2e68c78511a33ff62657d0292cac52a1d2. We have prod, stage and dev DNS to remove as well as VM decom.

Also, Zeus decom.
VM Decom is blocked on verification that product-details can also be decommed.

Summary: Please backup and decommission ship-it v1 → Please backup and decommission ship-it v1 and product-details

As per Slack discussion, adding product-details.mozilla.org in MDC1 to this decom as it's a sibling site on the same VM and has also been migrated to AWS.

product-details puppet removed in b9921e728b01aecfcc91ec0d2f411fcad4afd5eb

Moving to virtualization queue for the rest of decom process. Please decommission shipit1.stage.webapp.mdc1.mozilla.com, shipit1.webapp.mdc1.mozilla.com, shipit2.webapp.mdc1.mozilla.com VMs, associated Zeus configs for ship-it.mozilla.org and product-details.mozilla.org. Also, DNS for ship-it.mozilla.org. I'd be happy to help on those last few if desired. DNS for product-details has been updated to the new location and should remain untouched.

Assignee: eziegenhorn → server-ops-virtualization
Component: IT-Managed Tools → Virtualization

Just to make sure about the DNS changes:

  1. we don't use ship-it.mozilla.org anymore, feel free to remove it from the zone.
  2. product-details.mozilla.org DNS has been moved to Cloud Ops, no changes required to the zone.

OK, I've got the VMs - I can remove them, and their specific DNS - Eric: As to any other DNS, and ZLB, I'd appreciate the help if you can.

Running the server decom list:

shipit1.stage.webapp.mdc1.mozilla.com - 10.48.81.16
shipit1.webapp.mdc1.mozilla.com - 10.48.81.17
shipit2.webapp.mdc1.mozilla.com - This VM appears to have already been removed. No VM, No DNS.

puppet has been disabled.
VM removed from nagios checks.
VMs powered down. Waiting a week or so for screaming, then will continue with phase 2.

6 Zeus vservers disabled in MDC1 internal+external, will also wait until next week to delete everything there and DNS.

product-details-stage
[x] vserver
[x] pool
[x] TIG
[x] DNS

product-details-prod
[x] vserver
[x] pool
[x] TIG
[x] DNS for product-details.public.mdc1.mozilla.com removed
[x] DNS for public-details.mozilla.org CNAME is intact

ship-it-mdc1
[x] vserver
[x] pool,
[x] TIG,
[x] DNS removed for ship-it.public.mdc1.mozilla.com, ship-it-mdc1.mozilla.org removed

ship-it-internal TBD

Time for phase 2:
Alright, removed the specific DNS:
shipit1.stage.webapp.mdc1.mozilla.com - 10.48.81.16
shipit1.webapp.mdc1.mozilla.com - 10.48.81.17

removed from puppet, inventory, tracking, and deleted from vcenter.
pulled proxy rules as well.
bug 1564848 filed for netops to remove from panorama.

From my end this is now complete.

Whiteboard: [vm-delete:2]

10.48.87.21 - ship-it-stage.izlb.mdc1.mozilla.com
10.48.87.19 - ship-it.izlb.mdc1.mozilla.com
^ These still ping. Incomplete zeus cleanup?

10.48.75.71 - ship-it-stage.private.mdc1.mozilla.com.
10.48.75.50 - ship-it.private.mdc1.mozilla.com
^ These still exist, at least in DNS. Not sure how connected they are to the process, but they don't ping, which seems like a cleanup opportunity.

Wow! This is a lot of work! Thanks a lot.

(In reply to Greg Cox [:gcox] from comment #16)

10.48.87.21 - ship-it-stage.izlb.mdc1.mozilla.com
10.48.87.19 - ship-it.izlb.mdc1.mozilla.com
^ These still ping. Incomplete zeus cleanup?

10.48.75.71 - ship-it-stage.private.mdc1.mozilla.com.
10.48.75.50 - ship-it.private.mdc1.mozilla.com
^ These still exist, at least in DNS. Not sure how connected they are to the process, but they don't ping, which seems like a cleanup opportunity.

Yep, I haven't completed ship-it-internal bits for Zeus and DNS. (Noted in Comment 14)

ship-it-internal, ship-it-stage
[x] vserver
[x] pool,
[x] TIG,
[x] DNS removed for ship-it-stage.izlb.mdc1.mozilla.com, ship-it.allizom.org, ship-it.izlb.mdc1.mozilla.com, ship-it.mozilla.org

I've gone ahead and removed DNS for ship-it.private.mdc1 and ship-it-stage.private.mdc1 as well.

IMO this is good to close out.

Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED

Woot! Thank you very much!

You need to log in before you can comment on or make changes to this bug.