Closed Bug 1308496 Opened 3 years ago Closed 3 years ago

Decommission SCL3 Treeherder

Categories

(Infrastructure & Operations :: MOC: Service Requests, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Assigned: Usul)

References

Details

(Keywords: spring-cleaning, Whiteboard: [vm-delete:23])

Attachments

(1 file)

Treeherder moved from SCL3 to Heroku on 5th October (bug 1176484), retaining the same prod+stage domain names as before.

We're ready for:
1) all Nagios alerts (apart from the treeherder.{mozilla,allizom}.org SSL cert checks) to be permanently turned off
2) the SCL3 VMs to be powered off, and held for N days (where N is whatever is typical)

The VM instances are:
* treeherderadm.private.scl3.mozilla.com -> 10.22.75.150
* treeherder1.webapp.scl3.mozilla.com -> 10.22.81.138
* treeherder2.webapp.scl3.mozilla.com -> 10.22.81.160
* treeherder3.webapp.scl3.mozilla.com -> 10.22.81.36
* treeherder-processor1.stage.private.scl3.mozilla.com -> 10.22.75.22
* treeherder-processor2.stage.private.scl3.mozilla.com -> 10.22.75.23
* treeherder-processor3.stage.private.scl3.mozilla.com -> 10.22.75.24
* treeherder-processor1.private.scl3.mozilla.com -> 10.22.75.181
* treeherder-processor2.private.scl3.mozilla.com -> 10.22.75.183
* treeherder-processor3.private.scl3.mozilla.com -> 10.22.75.27
* treeherder-etl1.stage.private.scl3.mozilla.com -> 10.22.75.25
* treeherder-etl2.stage.private.scl3.mozilla.com -> 10.22.75.26
* treeherder-etl1.private.scl3.mozilla.com -> 10.22.75.28
* treeherder-etl2.private.scl3.mozilla.com -> 10.22.75.29
* treeherder-rabbitmq1.stage.private.scl3.mozilla.com -> 10.22.75.31
* treeherder-rabbitmq2.private.scl3.mozilla.com -> 10.22.75.37
* treeherder1.stage.db.scl3.mozilla.com -> 10.22.70.103
* treeherder2.stage.db.scl3.mozilla.com -> 10.22.70.104
* treeherder1.db.scl3.mozilla.com -> 10.22.70.136
* treeherder2.db.scl3.mozilla.com -> 10.22.70.137

After that holding period, we can remove:
* Puppet (manifests/nodes/treeherder.pp, modules/treeherder(_new)?/*, some of manifests/nodes/db_mysql/, ...)
* Hiera secrets (:fubar will need to confirm if any need to be kept for RDS/...)
* [Done in bug 1273916 comment 73] Proxy flows.
* The VPN/VPC/flows set up between SCL3 and AWS:
  - bug 1239660
  - bug 1285995
* The internal flows added in:
  - bug 1009142
  - bug 1015490
  - bug 1021889
  - bug 1025060
  - bug 1080590
  - bug 1100405
  - bug 1104952
  - bug 1113782
  - bug 1114709
  - bug 1118249
  - bug 1126226
  - bug 1128963
  - bug 1136284
  - bug 1142508
  - bug 1152976
  - bug 1260844
  - bug 1284876
* VIPs:
  - treeherder.vips.scl3.mozilla.com
  - treeherder.stage.vips.scl3.mozilla.com (or whatever stage was, I can't remember)
  - treeherder-ro-vip.metrics.scl3.mozilla.com
  - treeherder-rw-vip.metrics.scl3.mozilla.com
  - treeherder-stage-ro-vip.db.scl3.mozilla.com
  - treeherder-stage-rw-vip.db.scl3.mozilla.com
* Inventory entries
* Backups
* treeherder-dev.allizom.org DNS/... (Note *not* treeherder.allizom.org). (I believe it may have been missed after bug 1164023.)
* vpn_treeherder ACLs/LDAP group/...
* <whatever else is typical>

If there are any questions, :fubar will know more than I :-)

Many thanks!
Assignee: nobody → ludovic
(In reply to Ed Morley [:emorley] from comment #0)

> The VM instances are:

Missed the staging VMs:

> * treeherder1.stage.webapp.scl3.mozilla.com -> 10.22.81.29
> * treeherder2.stage.webapp.scl3.mozilla.com -> 10.22.81.30
> * treeherder3.stage.webapp.scl3.mozilla.com -> 10.22.81.31


> After that holding period, we can remove:
> * Puppet (manifests/nodes/treeherder.pp, modules/treeherder(_new)?/*, some
> of manifests/nodes/db_mysql/, ...)
> * Hiera secrets (:fubar will need to confirm if any need to be kept for
> RDS/...)

I'll handle the puppet clean up.

> * VIPs:
>   - treeherder.stage.vips.scl3.mozilla.com (or whatever stage was, I can't
> remember)

treeherder-stage.vips.scl3.mozilla.com
Keywords: spring-cleaning
removed from nagios in 122602.
no backups nor nfs.
puppet disabled

cleaned-up NewRelic
end phase One.
Duplicate of this bug: 1310832
inventory+dns cleaned up
Depends on: 1311330
cleanup puppet dashboard.
Attached file fubar.txt
git diff is not showing my /manifest/nodes/treeherder.pp rm.

Anything else that should go away fubar ?
Attachment #8802490 - Flags: review?(klibby)
Depends on: 1311344
Gcox, cknowles can you deleted the vms ?
OK, that's lots of VMs - starting.
Alright, the VMs mentioned in comments 0 and 1 have been deleted - the associated vApps have been removed, and they've been cleaned from our tracking sheets.
Whiteboard: [vm-delete:23]
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Comment on attachment 8802490 [details]
fubar.txt

You can also nuke modules/treeherder{,_new}, modules/webapp/files/treeherder{,-stage}, hiera/nodes/treeherder* in the puppet repo.

In the secrets repo, there's a bunch of stuff in hiera/site.yaml that can go.
Attachment #8802490 - Flags: review?(klibby) → review+
To ssh://gitolite3@git-internal.mozilla.org/sysadmins/puppet.git
   92f2018..827cc8f  master -> master


can you take care of secrets fubar ?
Flags: needinfo?(klibby)
Done. e5c5541..acd8e89  master -> master

In puppet, I see there's still a bunch of stuff in modules/nagios4. Not sure what the status of that module is, but maybe worth cleaning up while we're at it?
Flags: needinfo?(klibby)
Depends on: 1311493
I see a few more references to Treeherder left in the puppet repo:

$ grep -iIlr treeherder --exclude='vpn_treeherder.pp' .
./hiera/site.yaml
./manifests/nodes/db_mysql/databases.pp
./manifests/nodes/db_mysql/dbbackups.pp
./manifests/nodes/db_mysql/newrelic.pp
./modules/mysql2/files/backup_scripts/db-rawcopies
Dunno if the mysql stuff is normally handled by DBAs, but it seems safe to clean up. 

To ssh://git-internal.mozilla.org/sysadmins/puppet.git
   c3f0468..d35a0e2  master -> master
Depends on: 1313103
No longer depends on: 1313103
You need to log in before you can comment on or make changes to this bug.