Not sure how long ago, but pretty recently, vm1-1.labs.sjc1.mozilla.com lost all outbound connectivity. I can still reach it via it's internal address, but nothing goes out from it. This is a KVM host, and it's also affecting all VMs on it.
Affected VMs are: amqp1.vm1.labs.sjc1.mozilla.com has address 10.110.4.50 appdir1.vm1.labs.sjc1.mozilla.com has address 10.110.4.44 apprepo1.vm1.labs.sjc1.mozilla.com has address 10.110.4.43 apprunner1.vm1.labs.sjc1.mozilla.com has address 10.110.4.45 apps1.vm1.labs.sjc1.mozilla.com has address 10.110.4.47 appsite1.vm1.labs.sjc1.mozilla.com has address 10.110.4.42 backups1.vm1.labs.sjc1.mozilla.com has address 10.110.4.110 bbb1.vm1.labs.sjc1.mozilla.com has address 10.110.4.52 betafarm.vm1.labs.sjc1.mozilla.com has address 10.110.4.48 browserid-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.58 browserid-stats1.vm1.labs.sjc1.mozilla.com has address 10.110.4.66 catdv1.vm1.labs.sjc1.mozilla.com has address 10.110.4.108 htmlpad1.vm1.labs.sjc1.mozilla.com has address 10.110.4.25 ignite1.vm1.labs.sjc1.mozilla.com has address 10.110.4.73 journalism-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.102 lucid-lamp.vm1.labs.sjc1.mozilla.com has address 10.110.4.29 make-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.116 notredam1.vm1.labs.sjc1.mozilla.com has address 10.110.4.34 openbadges-stage1.vm1.labs.sjc1.mozilla.com has address 10.110.4.77 pancake-db-stage1.vm1.labs.sjc1.mozilla.com has address 10.110.4.128 pancake-hbase1.vm1.labs.sjc1.mozilla.com has address 10.110.4.68 pancake-hbase3.vm1.labs.sjc1.mozilla.com has address 10.110.4.70 pancake-thumb1.vm1.labs.sjc1.mozilla.com has address 10.110.4.126 persona-apps-prod1.vm1.labs.sjc1.mozilla.com has address 10.110.4.134 persona-dev1.vm1.labs.sjc1.mozilla.com has address 10.110.4.131 popcorn-stage1.vm1.labs.sjc1.mozilla.com has address 10.110.4.76 sift-ci1.vm1.labs.sjc1.mozilla.com has address 10.110.4.56 zabbix1.vm1.labs.sjc1.mozilla.com has address 10.110.4.122
16:53 <gozer> casey: 10.110.4.118 16:53 <gozer> is one I am trying 16:53 <gozer> casey: the VMs are on a different VLAN 16:54 <casey> fwsm broke your nats, i have to clear them by hand. so if you have a list, stick them in the bug 16:54 <casey> 118 should be live now 16:55 <gozer> casey: yup, works 16:55 <gozer> casey: interesting, it's happened before, but never to the VM host itself 16:56 <gozer> casey: I believe you could just clear the whole subnet, i recall ravi doing something like that before 16:57 <gozer> casey: in the bug, I listed all VMs on that host, they *might* not all be affected.. 16:58 --> casey kills the whole subnet all xlates have been cleared.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.