Closed Bug 1253390 Opened 9 years ago Closed 9 years ago

Decom FTP cluster

Categories

(Infrastructure & Operations :: Virtualization, task)

Product:
Infrastructure & Operations
Component:
Virtualization
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gcox, Assigned: gcox)

References

Details

(Keywords: spring-cleaning, Whiteboard: [vm-delete:10])

Assuming the VIPs stay down in bug 1239033, it'll be time to get rid of the ftp and upload clusters. Everything finally got shuttered today, so, slow-roll to this.
Disabled a few cron jobs that were still running on upload-cron.private.scl3, excluding those for /mnt/pvtbuilds. Committed revision 115368.
Audit before we get into this, because this is going to get MESSY when we actually start deleting. upload pool is: 10.22.74.51 = upload1.dmz.scl3 10.22.74.52 = upload2.dmz.scl3 10.22.75.128 = upload-cron.private.scl3 VS list: ZLB Servers: upload-ssh ZLB Traffic Group: upload-zlb.vips.scl3.mozilla.com:22 (63.245.215.47) ZLB Pools: upload1-ssh with upload2-ssh as a fail pool ZLB Servers: pvtbuilds.pvt.build.mozilla.org-ssh ZLB Traffic Group: pvtbuilds-zlb.dmz.scl3.mozilla.com:22 (10.22.74.161) ZLB Pools: upload1-ssh with upload2-ssh as a fail pool ftp pool is: 10.22.74.48 = ftp1.dmz.scl3 10.22.74.49 = ftp2.dmz.scl3 10.22.74.50 = ftp3.dmz.scl3 10.22.74.61 = ftp4.dmz.scl3 10.22.74.70 = ftp5.dmz.scl3 10.22.74.71 = ftp6.dmz.scl3 10.22.74.129 = ftp8.dmz.scl3 (there's no 7) Pools of the above are ftp-ftp (everyone on port 21), ftp-http (everyone on port 80), ftp-https (everyone on port 81) VS list: ZLB Servers: download-origin.cdn.mozilla.net-80 ZLB Traffic Group: download-origin-zlb.vips.scl3.mozilla.com:80 (63.245.215.86:80) ZLB Pools: ftp-http ZLB Servers: ftp-ftp ZLB Traffic Group: ftp1-zlb.vips.scl3.mozilla.com:21 (63.245.215.46:21, 63.245.215.56:21) ZLB Pools: ftp-ftp ZLB Servers: ftp-http ZLB Traffic Group: ftp1-zlb.vips.scl3.mozilla.com:80 (63.245.215.46:80, 63.245.215.56:80) Associated rules - drop-ms-bits, strip-nocache, fast-403-mar-files ZLB Pools: ftp-http ZLB Servers: ftp-https ZLB Traffic Group: ftp-ssl-zlb.vips.scl3.mozilla.com:443 (63.245.215.126:443, 63.245.215.127:443, 63.245.215.129:443) ftp1-zlb.vips.scl3.mozilla.com:443 (63.245.215.46:443, 63.245.215.56:443) Associated rules - drop-ms-bits, fast-403-mar-files, redirect-installers-to-cdn ZLB Pools: ftp-https ZLB Servers: partnerbuilds-http ZLB Traffic Group: partnerbuilds-zlb.vips.scl3.mozilla.com:80 (63.245.215.65:80) ZLB Pools: ftp-http ZLB Servers: partnerbuilds-https ZLB Traffic Group: partnerbuilds-zlb.vips.scl3.mozilla.com:443 (63.245.215.65:443) ZLB Pools: ftp-https ZLB Servers: pvtbuilds.pvt.build.mozilla.org-http ZLB Traffic Group: pvtbuilds-zlb.dmz.scl3.mozilla.com:80 (10.22.74.161:80) Associated rules - Forwarded-for, 'robots.txt - pvtbuilds.mozilla.org' ZLB Pools: ftp-http ZLB Servers: pvtbuilds.pvt.build.mozilla.org-https ZLB Traffic Group: pvtblds-zlb.vips.scl3.mozilla.com:443 (63.245.215.32:443 , name in zeus doesn't match DNS) pvtbuilds-zlb.dmz.scl3.mozilla.com:443 (10.22.74.161:443) Associated rules - Forwarded-for, 'robots.txt - pvtbuilds.mozilla.org' ZLB Pools: ftp-https ZLB Servers: releases-http ZLB Traffic Group: releases-zlb.vips.scl3.mozilla.com:80 (63.245.215.63:80) Associated rules - Forwarded-for ZLB Pools: ftp-http ZLB Servers: releases-https ZLB Traffic Group: releases-zlb.vips.scl3.mozilla.com:443 (63.245.215.63:443) Associated rules - Forwarded-for ZLB Pools: ftp-https ZLB Servers: upload-http ZLB Traffic Group: upload-zlb.vips.scl3.mozilla.com:80 (63.245.215.47:80) Associated rules - redirect-stage.mozilla.org ZLB Pools: ftp-http
Distilling the above into actual IP and name touches with less mess: 10.22.74.51 = upload1.dmz.scl3 10.22.74.52 = upload2.dmz.scl3 10.22.75.128 = upload-cron.private.scl3 10.22.74.48 = ftp1.dmz.scl3 10.22.74.49 = ftp2.dmz.scl3 10.22.74.50 = ftp3.dmz.scl3 10.22.74.61 = ftp4.dmz.scl3 10.22.74.70 = ftp5.dmz.scl3 10.22.74.71 = ftp6.dmz.scl3 10.22.74.129 = ftp8.dmz.scl3 10.22.74.161 = pvtbuilds-zlb.dmz.scl3 (and CNAME pvtbuilds.pvt.build.mozilla.org) 63.245.215.32 = pvtbuilds-zlb.vips.scl3 (and A record pvtbuilds.mozilla.org) 63.245.215.46 = ftp1-zlb.vips.scl3 (and A records ftp-origin-scl3.mozilla.org, ftp-scl3.mozilla.com, ftp-zlb.vips.scl3) (and nested CNAME download02.mozilla.org -> dm-download02.mozilla.org -> ftp-zlb.vips.scl3) 63.245.215.47 = upload-zlb.vips.scl3 (and CNAMEs china-sync.mozilla.org, pv-mirror01.mozilla.org, pv-mirror02.mozilla.org -> upload-zlb.vips.scl3) (and nested CNAMEs (partner.mozilla.org, platform.mozilla.org, stage-old.mozilla.org) -> surf.mozilla.org -> upload-zlb.vips.scl3) 63.245.215.56 = ftp2-zlb.vips.scl3 (and A record ftp-origin-scl3.mozilla.org) 63.245.215.63 = releases-zlb.vips.scl3 63.245.215.65 = partnerbuilds-zlb.vips.scl3 (and A record partnerbuilds.mozilla.com) 63.245.215.86 = download-origin-zlb.vips.scl3 (and CNAME download-origin-scl3.cdn.mozilla.net) 63.245.215.126 = ftp-ssl.mozilla.org (but a mismatched A of ftp-ssl-zlb.vips.scl3). ftp-ssl.mozilla.org is a CNAME to cloudfront. 63.245.215.127 = ftp-ssl.mozilla.org (but a mismatched A of ftp-ssl-zlb.vips.scl3). ftp-ssl.mozilla.org is a CNAME to cloudfront. 63.245.215.129 = ftp-ssl.mozilla.org (but a mismatched A of ftp-ssl-zlb.vips.scl3). ftp-ssl.mozilla.org is a CNAME to cloudfront.
Externals were already purged. Internal IPs removed from nagios. sysadmins repo svn diff -r 115541:115543
Complexity reduction: decom'ed ftp2-8 and upload2 (did not purge puppet yet) This just clears out overbuilt pools, for now. Left one each of ftp, upload, upload-cron in existence (for now). Powered those VMs off, though, which cuts off all VMs that have access to the stage volume.
Whiteboard: [vm-delete:7]
Having sat for a long while, it's now time. Deleted rhn, puppetdashboard, dns, inventory, newrelic for final VMs. Cleaned up filer ACLs. This removes all users of ffxbld and ftp_stage, which I will delete this week. Zeus: cleared up items from comment 2, followed by the DNS entries from comment 3. Took a whack out of puppet: Sending manifests/nodes/productdelivery.pp Deleting modules/productdelivery/files/bin/archive-debug-builds.sh Deleting modules/productdelivery/files/bin/extract_and_run_command.py Deleting modules/productdelivery/files/bin/ftp-trim-links.sh Deleting modules/productdelivery/files/bin/get-rsync-module-size Deleting modules/productdelivery/files/bin/mar.py Deleting modules/productdelivery/files/bin/motd-gen.sh Deleting modules/productdelivery/files/bin/rsync_files.sh Deleting modules/productdelivery/files/bin/scan-file.sh Deleting modules/productdelivery/files/bin/scan-uploaded-files.sh Deleting modules/productdelivery/files/bin/stage-make-nightlydirs Deleting modules/productdelivery/files/bin/symlink-latest-release.py Deleting modules/productdelivery/files/bin/trim-lightning-nightlies.py Deleting modules/productdelivery/files/bin/trim-thunderbird-nightlies.py Deleting modules/productdelivery/files/bin/validate-rsync Deleting modules/productdelivery/files/cron Deleting modules/productdelivery/files/cron.d Deleting modules/productdelivery/files/etc/clamd.conf Deleting modules/productdelivery/files/etc-httpd Deleting modules/productdelivery/files/rsync Deleting modules/productdelivery/files/users.d Deleting modules/productdelivery/files/vsftpd Deleting modules/productdelivery/manifests/ftp.pp Deleting modules/productdelivery/manifests/mounts/pvtbuilds_rw.pp Deleting modules/productdelivery/manifests/pvtbuilds_cron.pp Deleting modules/productdelivery/manifests/upload.pp Deleting modules/productdelivery/manifests/upload_cron.pp Deleting modules/secrets/files/productdelivery Committed revision 115968. 3 more VMs deleted. Will file netops and VPN cleanup bugs Will follow up with filer volume deletes.
Whiteboard: [vm-delete:7] → [vm-delete:10]
The FTP vols have been deleted from the filer. That's it.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.