vm1-1.labs.sjc1.mozilla.com ( lost access to the world



Infrastructure & Operations
6 years ago
5 years ago


(Reporter: gozer, Unassigned)





6 years ago
Not sure how long ago, but pretty recently, vm1-1.labs.sjc1.mozilla.com lost all outbound connectivity. I can still reach it via it's internal address, but nothing goes out from it.

This is a KVM host, and it's also affecting all VMs on it.

Comment 1

6 years ago
Affected VMs are:

amqp1.vm1.labs.sjc1.mozilla.com has address
appdir1.vm1.labs.sjc1.mozilla.com has address
apprepo1.vm1.labs.sjc1.mozilla.com has address
apprunner1.vm1.labs.sjc1.mozilla.com has address
apps1.vm1.labs.sjc1.mozilla.com has address
appsite1.vm1.labs.sjc1.mozilla.com has address
backups1.vm1.labs.sjc1.mozilla.com has address
bbb1.vm1.labs.sjc1.mozilla.com has address
betafarm.vm1.labs.sjc1.mozilla.com has address
browserid-dev1.vm1.labs.sjc1.mozilla.com has address
browserid-stats1.vm1.labs.sjc1.mozilla.com has address
catdv1.vm1.labs.sjc1.mozilla.com has address
htmlpad1.vm1.labs.sjc1.mozilla.com has address
ignite1.vm1.labs.sjc1.mozilla.com has address
journalism-dev1.vm1.labs.sjc1.mozilla.com has address
lucid-lamp.vm1.labs.sjc1.mozilla.com has address
make-dev1.vm1.labs.sjc1.mozilla.com has address
notredam1.vm1.labs.sjc1.mozilla.com has address
openbadges-stage1.vm1.labs.sjc1.mozilla.com has address
pancake-db-stage1.vm1.labs.sjc1.mozilla.com has address
pancake-hbase1.vm1.labs.sjc1.mozilla.com has address
pancake-hbase3.vm1.labs.sjc1.mozilla.com has address
pancake-thumb1.vm1.labs.sjc1.mozilla.com has address
persona-apps-prod1.vm1.labs.sjc1.mozilla.com has address
persona-dev1.vm1.labs.sjc1.mozilla.com has address
popcorn-stage1.vm1.labs.sjc1.mozilla.com has address
sift-ci1.vm1.labs.sjc1.mozilla.com has address
zabbix1.vm1.labs.sjc1.mozilla.com has address
16:53 <gozer> casey:
16:53 <gozer> is one I am trying
16:53 <gozer> casey: the VMs are on a different VLAN
16:54 <casey> fwsm broke your nats, i have to clear them by hand. so if you have a list, stick them in the bug
16:54 <casey> 118 should be live now
16:55 <gozer> casey: yup, works
16:55 <gozer> casey: interesting, it's happened before, but never to the VM host itself
16:56 <gozer> casey: I believe you could just clear the whole subnet, i recall ravi doing something like that before
16:57 <gozer> casey: in the bug, I listed all VMs on that host, they *might* not all be affected..
16:58  --> casey kills the whole subnet

all xlates have been cleared.
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.