791337 - Investigate/Evaluate Zeus config improvements

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Jake Maul [:jakem]]

Something to discuss at the October meet-up, and hopefully nail down some more concrete plans.

Some ideas so far:

1) Unify naming schemes of TIGs, vservers, and pools (as well as SSL certs, rules, and anything else with a name/description field). This is easy but tedious. Let's hash out a naming scheme at the meet-up. At a glance, this seems feasible to actually have completed in Q4.

2) Merger of :80 and :81 pools. New policy is apparently that everything that *can* be SSL, should be. Many :80 pools could possibly be eliminated and replaced by a trafficscript that does the redirect. In other cases where HTTP is still needed, it might still be possible to combine them and put a couple lines in the Apache config to help differentiate "secured" vs "unsecured" traffic for the app. In the end, we should probably have very few :80 pools left.

3) Merger of unnecessary duplicate vservers. For example, the intranet cluster has 5 vservers in PHX1 (intranet, inventory, pto, ldap-prod, phonebook)... seems likely that 1 would do just as well.

4) Merger/Reorg of TIGs. IPs within a TIG are automatically distributed across all available LBs. Vservers can bind to specific IPs rather than whole TIGs. For example, we could put all WebOps stuff in one TIG, AMO Ops in another, Infra in another, etc... then list individual IPs in each vserver. Other schemes might be better, but the basic idea is that it's inefficient to have one IP per TIG... it creates extra TIGs and requires manual IP distribution among LBs. Any services that are big enough to cause concern wrt overloading an LB should not be isolated on an LB- they should get additional IPs and be *spread across* LBs via DNS, or split out into a dedicated Zeus cluster.

5) Finalize migration of internal services to internal Zeus cluster.

6) Unify settings and naming schemes across Zeus clusters.


... plus whatever other changes could be made that would make Zeus "better" (easier to use, more efficient, more scalable, more obvious, etc). Open for discussion.

Marking as P4 for how, should be bumped to P1 when Q4 starts. Also marking this as a [tracker] bug... concrete plans should get dependent bugs.

Comment 1

[Jake Maul [:jakem]]

It turns out #4 is not actually possible. I tried it, failed, and confirmed with Zeus that it doesn't work like that.

#1 and #6 are functionally the same.

Marking a dependent bug about fixing some networking layout stuff w.r.t. internal Zeus nodes in PHX1.

Etherpad link from discussions at webops-q4 work week: https://infra.etherpad.mozilla.org/webopsww-zeus

Comment 2

[Jake Maul [:jakem]]

Missed this as a Q4 goal... it's not a Q1 goal, so removing the tag.