1382858 - Mac tests have started permafailing with FATAL ERROR: Non-local network connections are disabled and a connection attempt to example (52.40.50.201) was made.

Status: RESOLVED FIXED

(Firefox :: Address Bar, enhancement)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Various tests are suddenly permafailing like https://treeherder.mozilla.org/logviewer.html#?job_id=116104969&repo=autoland on various trees.

Inbound has ones that look like https://treeherder.mozilla.org/logviewer.html#?job_id=116150967&repo=mozilla-inbound

ESR52 has ones that look like https://treeherder.mozilla.org/logviewer.html#?job_id=116138094&repo=mozilla-esr52

I'm going to be closing trees.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This might be new ?

$ host example
example.mozilla.org is an alias for www.haul-stage.stage.us-west-2.appsvcs-static.nubis.allizom.org.
www.haul-stage.stage.us-west-2.appsvcs-static.nubis.allizom.org is an alias for haul-web-stage-1545714456.us-west-2.elb.amazonaws.com.
haul-web-stage-1545714456.us-west-2.elb.amazonaws.com has address 52.40.50.201
haul-web-stage-1545714456.us-west-2.elb.amazonaws.com has address 52.34.31.151

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

That was run on t-yosemite-r7-0231.test.releng.scl3.mozilla.com, and matches the IPs in the logs Wes gave:

13:05:39     INFO - GECKO(1730) | FATAL ERROR: Non-local network connections are disabled and a connection attempt to example (52.40.50.201) was made.

14:51:16     INFO - GECKO(4274) | FATAL ERROR: Non-local network connections are disabled and a connection attempt to example (52.34.31.151) was made.

14:15:19     INFO - FATAL ERROR: Non-local network connections are disabled and a connection attempt to example (52.40.50.201) was made.

Comment 3

[Daniel Hartnell [:danielh]]

I have removed the CNAME for example.mozilla.org to www.haul-stage.stage.us-west-2.appsvcs-static.nubis.allizom.org. It had a 60 second TTL.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Now seeing

$ host example
Host example not found: 3(NXDOMAIN)


On the bigger picture, how can we prevent this in future ? Could the test be made more robust ?

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Reopened trees.

Comment 6

[Intermittent Failures Robot]

146 failures in 822 pushes (0.178 failures/push) were associated with this bug in the last 7 days. 

This is the #8 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 103
* mozilla-inbound: 43

Platform breakdown:
* osx-10-10: 146

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1382858&startday=2017-07-17&endday=2017-07-23&tree=all