Status

Infrastructure & Operations
Virtualization
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: gcox, Assigned: gcox)

Tracking

({spring-cleaning})

Details

(Whiteboard: [vm-delete:10])

(Assignee)

Description

2 years ago
Assuming the VIPs stay down in bug 1239033, it'll be time to get rid of the ftp and upload clusters.

Everything finally got shuttered today, so, slow-roll to this.
Disabled a few cron jobs that were still running on upload-cron.private.scl3, excluding those for /mnt/pvtbuilds.

Committed revision 115368.
(Assignee)

Comment 2

2 years ago
Audit before we get into this, because this is going to get MESSY when we actually start deleting.
upload pool is:
10.22.74.51  = upload1.dmz.scl3
10.22.74.52  = upload2.dmz.scl3
10.22.75.128 = upload-cron.private.scl3

VS list:
ZLB Servers: upload-ssh
ZLB Traffic Group:  upload-zlb.vips.scl3.mozilla.com:22 (63.245.215.47)
ZLB Pools: upload1-ssh with upload2-ssh as a fail pool

ZLB Servers: pvtbuilds.pvt.build.mozilla.org-ssh
ZLB Traffic Group:  pvtbuilds-zlb.dmz.scl3.mozilla.com:22 (10.22.74.161)
ZLB Pools: upload1-ssh with upload2-ssh as a fail pool


ftp pool is:
10.22.74.48  = ftp1.dmz.scl3
10.22.74.49  = ftp2.dmz.scl3
10.22.74.50  = ftp3.dmz.scl3
10.22.74.61  = ftp4.dmz.scl3
10.22.74.70  = ftp5.dmz.scl3
10.22.74.71  = ftp6.dmz.scl3
10.22.74.129 = ftp8.dmz.scl3
(there's no 7)

Pools of the above are ftp-ftp (everyone on port 21), ftp-http (everyone on port 80), ftp-https (everyone on port 81)
VS list:

ZLB Servers: download-origin.cdn.mozilla.net-80
ZLB Traffic Group: download-origin-zlb.vips.scl3.mozilla.com:80 (63.245.215.86:80)
ZLB Pools: ftp-http

ZLB Servers: ftp-ftp
ZLB Traffic Group: ftp1-zlb.vips.scl3.mozilla.com:21 (63.245.215.46:21, 63.245.215.56:21)
ZLB Pools: ftp-ftp

ZLB Servers: ftp-http
ZLB Traffic Group: ftp1-zlb.vips.scl3.mozilla.com:80 (63.245.215.46:80, 63.245.215.56:80)
  Associated rules - drop-ms-bits, strip-nocache, fast-403-mar-files
ZLB Pools: ftp-http

ZLB Servers: ftp-https
ZLB Traffic Group:  ftp-ssl-zlb.vips.scl3.mozilla.com:443 (63.245.215.126:443, 63.245.215.127:443, 63.245.215.129:443)  ftp1-zlb.vips.scl3.mozilla.com:443 (63.245.215.46:443, 63.245.215.56:443)
  Associated rules - drop-ms-bits, fast-403-mar-files, redirect-installers-to-cdn
ZLB Pools: ftp-https

ZLB Servers: partnerbuilds-http
ZLB Traffic Group: partnerbuilds-zlb.vips.scl3.mozilla.com:80 (63.245.215.65:80)
ZLB Pools: ftp-http

ZLB Servers: partnerbuilds-https
ZLB Traffic Group: partnerbuilds-zlb.vips.scl3.mozilla.com:443 (63.245.215.65:443)
ZLB Pools: ftp-https

ZLB Servers: pvtbuilds.pvt.build.mozilla.org-http
ZLB Traffic Group: pvtbuilds-zlb.dmz.scl3.mozilla.com:80 (10.22.74.161:80)
  Associated rules - Forwarded-for, 'robots.txt - pvtbuilds.mozilla.org'
ZLB Pools: ftp-http

ZLB Servers: pvtbuilds.pvt.build.mozilla.org-https
ZLB Traffic Group: pvtblds-zlb.vips.scl3.mozilla.com:443 (63.245.215.32:443 , name in zeus doesn't match DNS) pvtbuilds-zlb.dmz.scl3.mozilla.com:443 (10.22.74.161:443)
  Associated rules - Forwarded-for, 'robots.txt - pvtbuilds.mozilla.org'
ZLB Pools: ftp-https

ZLB Servers: releases-http
ZLB Traffic Group: releases-zlb.vips.scl3.mozilla.com:80 (63.245.215.63:80)
  Associated rules - Forwarded-for
ZLB Pools: ftp-http

ZLB Servers: releases-https
ZLB Traffic Group: releases-zlb.vips.scl3.mozilla.com:443 (63.245.215.63:443)
  Associated rules - Forwarded-for
ZLB Pools: ftp-https

ZLB Servers: upload-http
ZLB Traffic Group: upload-zlb.vips.scl3.mozilla.com:80 (63.245.215.47:80)
  Associated rules - redirect-stage.mozilla.org
ZLB Pools: ftp-http
(Assignee)

Comment 3

2 years ago
Distilling the above into actual IP and name touches with less mess:

10.22.74.51    = upload1.dmz.scl3
10.22.74.52    = upload2.dmz.scl3
10.22.75.128   = upload-cron.private.scl3
10.22.74.48    = ftp1.dmz.scl3
10.22.74.49    = ftp2.dmz.scl3
10.22.74.50    = ftp3.dmz.scl3
10.22.74.61    = ftp4.dmz.scl3
10.22.74.70    = ftp5.dmz.scl3
10.22.74.71    = ftp6.dmz.scl3
10.22.74.129   = ftp8.dmz.scl3

10.22.74.161   = pvtbuilds-zlb.dmz.scl3 (and CNAME pvtbuilds.pvt.build.mozilla.org)
63.245.215.32  = pvtbuilds-zlb.vips.scl3 (and A record pvtbuilds.mozilla.org)
63.245.215.46  = ftp1-zlb.vips.scl3 (and A records ftp-origin-scl3.mozilla.org, ftp-scl3.mozilla.com, ftp-zlb.vips.scl3) (and nested CNAME download02.mozilla.org -> dm-download02.mozilla.org -> ftp-zlb.vips.scl3)
63.245.215.47  = upload-zlb.vips.scl3 (and CNAMEs china-sync.mozilla.org, pv-mirror01.mozilla.org, pv-mirror02.mozilla.org -> upload-zlb.vips.scl3) (and nested CNAMEs (partner.mozilla.org, platform.mozilla.org, stage-old.mozilla.org) -> surf.mozilla.org -> upload-zlb.vips.scl3)
63.245.215.56  = ftp2-zlb.vips.scl3 (and A record ftp-origin-scl3.mozilla.org)
63.245.215.63  = releases-zlb.vips.scl3
63.245.215.65  = partnerbuilds-zlb.vips.scl3 (and A record partnerbuilds.mozilla.com)
63.245.215.86  = download-origin-zlb.vips.scl3 (and CNAME download-origin-scl3.cdn.mozilla.net)
63.245.215.126 = ftp-ssl.mozilla.org (but a mismatched A of ftp-ssl-zlb.vips.scl3).  ftp-ssl.mozilla.org is a CNAME to cloudfront.
63.245.215.127 = ftp-ssl.mozilla.org (but a mismatched A of ftp-ssl-zlb.vips.scl3).  ftp-ssl.mozilla.org is a CNAME to cloudfront.
63.245.215.129 = ftp-ssl.mozilla.org (but a mismatched A of ftp-ssl-zlb.vips.scl3).  ftp-ssl.mozilla.org is a CNAME to cloudfront.
(Assignee)

Comment 4

2 years ago
Externals were already purged.  Internal IPs removed from nagios.
  sysadmins repo svn diff -r 115541:115543
(Assignee)

Updated

2 years ago
Depends on: 1256465
(Assignee)

Comment 5

2 years ago
Complexity reduction: decom'ed ftp2-8 and upload2 (did not purge puppet yet)
This just clears out overbuilt pools, for now.

Left one each of ftp, upload, upload-cron in existence (for now).
Powered those VMs off, though, which cuts off all VMs that have access to the stage volume.
(Assignee)

Updated

2 years ago
Depends on: 1257310
(Assignee)

Updated

2 years ago
Whiteboard: [vm-delete:7]
(Assignee)

Comment 6

2 years ago
Having sat for a long while, it's now time.

Deleted rhn, puppetdashboard, dns, inventory, newrelic for final VMs.

Cleaned up filer ACLs.  This removes all users of ffxbld and ftp_stage, which I will delete this week.
Zeus: cleared up items from comment 2, followed by the DNS entries from comment 3.

Took a whack out of puppet:
Sending        manifests/nodes/productdelivery.pp
Deleting       modules/productdelivery/files/bin/archive-debug-builds.sh
Deleting       modules/productdelivery/files/bin/extract_and_run_command.py
Deleting       modules/productdelivery/files/bin/ftp-trim-links.sh
Deleting       modules/productdelivery/files/bin/get-rsync-module-size
Deleting       modules/productdelivery/files/bin/mar.py
Deleting       modules/productdelivery/files/bin/motd-gen.sh
Deleting       modules/productdelivery/files/bin/rsync_files.sh
Deleting       modules/productdelivery/files/bin/scan-file.sh
Deleting       modules/productdelivery/files/bin/scan-uploaded-files.sh
Deleting       modules/productdelivery/files/bin/stage-make-nightlydirs
Deleting       modules/productdelivery/files/bin/symlink-latest-release.py
Deleting       modules/productdelivery/files/bin/trim-lightning-nightlies.py
Deleting       modules/productdelivery/files/bin/trim-thunderbird-nightlies.py
Deleting       modules/productdelivery/files/bin/validate-rsync
Deleting       modules/productdelivery/files/cron
Deleting       modules/productdelivery/files/cron.d
Deleting       modules/productdelivery/files/etc/clamd.conf
Deleting       modules/productdelivery/files/etc-httpd
Deleting       modules/productdelivery/files/rsync
Deleting       modules/productdelivery/files/users.d
Deleting       modules/productdelivery/files/vsftpd
Deleting       modules/productdelivery/manifests/ftp.pp
Deleting       modules/productdelivery/manifests/mounts/pvtbuilds_rw.pp
Deleting       modules/productdelivery/manifests/pvtbuilds_cron.pp
Deleting       modules/productdelivery/manifests/upload.pp
Deleting       modules/productdelivery/manifests/upload_cron.pp
Deleting       modules/secrets/files/productdelivery
Committed revision 115968.

3 more VMs deleted.

Will file netops and VPN cleanup bugs
Will follow up with filer volume deletes.
Whiteboard: [vm-delete:7] → [vm-delete:10]
(Assignee)

Updated

2 years ago
Depends on: 1260132
(Assignee)

Updated

2 years ago
Depends on: 1260133
(Assignee)

Comment 7

2 years ago
The FTP vols have been deleted from the filer.  That's it.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.