725404 - openwebapps-test1.vm1.labs.sjc1.mozilla.com (10.110.4.81) can't make outbound network connections anymore

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Philippe M. Chiasson (:gozer)]

This has happened before to other labs VM, and the issue was with some stuck NAT state on the firewall that needed unsticking.

Comment 1

[casey ransom [:casey]]

I applied the necessary solvent to unstick the nat translation and traffic is flowing again.

Comment 2

[[e-mail] [:onecyrenus]]

this is not fixed.. this will happen again.

Comment 3

[[e-mail] [:onecyrenus]]

someone needs to guarantee this won't happen every 2 months or so.. as my server silently fails

Comment 4

[casey ransom [:casey]]

It will not be an issue when the gear at SJC1 is replaced or migrated in SCL3. In the mean time, we cannot proactively detect these failures and replacing the hardware is not a viable option since SJC1 has a very limited lifespan.  SCL3 is the fix and it's being worked on feverishly right now.

Comment 5

[[e-mail] [:onecyrenus]]

Could you surface details about the requisite bug #, and what is the limited lifespan ? That seems fair to surface that information.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

We're slated to be out of sjc1 by May 1.  We're all sweating to get that move done.  The firewalls in sjc1 are poor, and we're not replicating that architecture in the new datacenter, so we don't expect this kind of problem after the move.

Gozer can tell you more about the move plan for labs systems in sjc1.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

(bug 716047 is the move tracker, although apologies that a lot of its dependents are infra-only, since there's a lot of infra involved!)