887304 - Internal zlb nodes in SCL3 need NICs upgraded to 10G

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Corey Shields [:cshields]]

zlb2 and zlb4 in SCL3 handle our "internal" traffic.  Recently some changes were made that cause excessive load on zlb2's 1G NICs, also causing bug 883018.

These nodes both have 10G interfaces in them (as the rest of the external-facing zlb nodes do).  We need to have them cabled and move to using them.

Given that when we move TIG's between zlb nodes for this work it causes a few seconds of downtime, this should be done during a tree-closing window to avoid affecting development.

Comment 1

[Derek Moore [:dmoore]]

zlb2 and zlb4 were intentionally left on 1g per Bug 758062. I just wanted to make sure the reasons for this decision were no longer relevant.

Also, as part of this work, we'll need to relocate these two servers to an adjacent rack with 10G switches.

Comment 2

[Ravi Pina [:ravi]]

Yeah, there is plenty of 10G capacity in rack 10 of both rows.

Comment 3

[Michal Purzynski [:michal`] (use NEEDINFO)]

Please also keep in mind that we're mirroring the traffic to the NSM so if a switch port is changed, a configuration will need to be updated. Just FYI.

Comment 4

[Corey Shields [:cshields]]

(In reply to Derek Moore from comment #1)
> zlb2 and zlb4 were intentionally left on 1g per Bug 758062. I just wanted to
> make sure the reasons for this decision were no longer relevant.
> 
> Also, as part of this work, we'll need to relocate these two servers to an
> adjacent rack with 10G switches.

So..  now I'm totally confused.  In that bug we say to skip zlb2 and zlb4, yet they have 10g interfaces.

Looking further, those 10g interfaces appear to be NetXens..   ick.  So..  do we have Myricoms set aside for this or do I need to order some?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

bug 889850 is the CAB bug for this.

Comment 6

[Shyam Mani [:fox2mike]]

Thanks to dmoore and jd, this is complete. We switched both zlb2 and zlb4 to 10G cards.