vm1-1.labs.sjc1.mozilla.com (10.2.110.41) lost access to the world

RESOLVED FIXED

Status

--
critical
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: gozer, Unassigned)

Tracking

Details

(Reporter)

Description

7 years ago
Not sure how long ago, but pretty recently, vm1-1.labs.sjc1.mozilla.com lost all outbound connectivity. I can still reach it via it's internal address, but nothing goes out from it.

This is a KVM host, and it's also affecting all VMs on it.
(Reporter)

Comment 1

7 years ago
Affected VMs are:

amqp1.vm1.labs.sjc1.mozilla.com has address 10.110.4.50
appdir1.vm1.labs.sjc1.mozilla.com has address 10.110.4.44
apprepo1.vm1.labs.sjc1.mozilla.com has address 10.110.4.43
apprunner1.vm1.labs.sjc1.mozilla.com has address 10.110.4.45
apps1.vm1.labs.sjc1.mozilla.com has address 10.110.4.47
appsite1.vm1.labs.sjc1.mozilla.com has address 10.110.4.42
backups1.vm1.labs.sjc1.mozilla.com has address 10.110.4.110
bbb1.vm1.labs.sjc1.mozilla.com has address 10.110.4.52
betafarm.vm1.labs.sjc1.mozilla.com has address 10.110.4.48
browserid-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.58
browserid-stats1.vm1.labs.sjc1.mozilla.com has address 10.110.4.66
catdv1.vm1.labs.sjc1.mozilla.com has address 10.110.4.108
htmlpad1.vm1.labs.sjc1.mozilla.com has address 10.110.4.25
ignite1.vm1.labs.sjc1.mozilla.com has address 10.110.4.73
journalism-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.102
lucid-lamp.vm1.labs.sjc1.mozilla.com has address 10.110.4.29
make-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.116
notredam1.vm1.labs.sjc1.mozilla.com has address 10.110.4.34
openbadges-stage1.vm1.labs.sjc1.mozilla.com has address 10.110.4.77
pancake-db-stage1.vm1.labs.sjc1.mozilla.com has address 10.110.4.128
pancake-hbase1.vm1.labs.sjc1.mozilla.com has address 10.110.4.68
pancake-hbase3.vm1.labs.sjc1.mozilla.com has address 10.110.4.70
pancake-thumb1.vm1.labs.sjc1.mozilla.com has address 10.110.4.126
persona-apps-prod1.vm1.labs.sjc1.mozilla.com has address 10.110.4.134
persona-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.131
popcorn-stage1.vm1.labs.sjc1.mozilla.com has address 10.110.4.76
sift-ci1.vm1.labs.sjc1.mozilla.com has address 10.110.4.56
zabbix1.vm1.labs.sjc1.mozilla.com has address 10.110.4.122
16:53 <gozer> casey: 10.110.4.118
16:53 <gozer> is one I am trying
16:53 <gozer> casey: the VMs are on a different VLAN
16:54 <casey> fwsm broke your nats, i have to clear them by hand. so if you have a list, stick them in the bug
16:54 <casey> 118 should be live now
16:55 <gozer> casey: yup, works
16:55 <gozer> casey: interesting, it's happened before, but never to the VM host itself
16:56 <gozer> casey: I believe you could just clear the whole subnet, i recall ravi doing something like that before
16:57 <gozer> casey: in the bug, I listed all VMs on that host, they *might* not all be affected..
16:58  --> casey kills the whole subnet



all xlates have been cleared.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.