1221014 - Intermittent [taskcluster:error] Task was aborted because states could not be created successfully. Error: connect EHOSTUNREACH

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Carsten Book [:Tomcat]]

https://treeherder.mozilla.org/logviewer.html#?job_id=2587365&repo=mozilla-central

askcluster:error] Task was aborted because states could not be created successfully. Error: connect EHOSTUNREACH
     at /home/ubuntu/docker_worker/lib/states.js:63:15
     at runMicrotasksCallback (node.js:337:7)
     at process._tickDomainCallback (node.js:381:11)

 [taskcluster] Unsuccessful task run with exit code: -1 completed in 181.471 seconds

Comment 1

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

The only feature enabled is "balrogVPNProxy" (see https://tools.taskcluster.net/task-inspector/#NRq1o6ECSxS1Cdm7-0NL6w/) so it is likely to be a problem with this.

Comment 2

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

We should probably update the docker worker error message to say which state(s) could not be created, and which features they relate to...

Comment 3

[Greg Arndt [:garndt]]

Cause of this is from bug 1220252 where the balrog vpn proxy was disabled.  Unfortunately the tasks have not been unscheduled.

Comment 4

[Carsten Book [:Tomcat]]

(In reply to Greg Arndt [:garndt] from comment #3)
> Cause of this is from bug 1220252 where the balrog vpn proxy was disabled. 
> Unfortunately the tasks have not been unscheduled.

can we unschedulde this task since this cause a lot of "failures" on treeherder ?

Comment 5

[Intermittent Failures Robot]

280 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 194
* mozilla-central: 86

Platform breakdown:
* linux32: 91
* osx-10-10: 66
* linux64: 63
* windows8-64: 57
* windowsxp: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1221014&startday=2015-11-03&endday=2015-11-03&tree=all

Comment 6

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

(In reply to Pete Moore [:pmoore][:pete] from comment #2)
> We should probably update the docker worker error message to say which
> state(s) could not be created, and which features they relate to...

Created bug 1221491 for this.

Comment 7

[Intermittent Failures Robot]

280 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 194
* mozilla-central: 86

Platform breakdown:
* linux32: 91
* osx-10-10: 66
* linux64: 63
* windows8-64: 57
* windowsxp: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1221014&startday=2015-11-02&endday=2015-11-08&tree=all

Comment 8

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

Looking good - all the above failures were on 3 November, and today is 9 November, so at least 5 days with no reports. Let's leave this open until the end of the month, and then close it if no more failures...

Comment 9

[Greg Arndt [:garndt]]

I am not sure why we would have any failures at least with this error.  This was specifically because it was disabled, but has since been re-enabled.  I don't have a strong opinion about keeping it open, but I don't expect this to happen again.

Comment 10

[Intermittent Failures Robot]

14 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 13
* mozilla-b2g44_v2_5: 1

Platform breakdown:
* osx-10-10: 5
* windowsxp: 2
* windows8-64: 2
* linux64: 2
* linux32: 2
* b2g-device-image: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1221014&startday=2015-11-23&endday=2015-11-29&tree=all

Comment 11

[Greg Arndt [:garndt]]

Looking at the last 14 jobs that have been starred, it seems two had this error signature.  Other errors that I see starred are:

I see errors like:
HTTPError: 400 Client Error: BAD REQUEST 
ssl.SSLError: ('The read operation timed out',) 
2015-11-25 14:22:19,875 - INFO - retry: Giving up on <function <lambda> at 0x7f26f571ca28> 
[taskcluster:error] Task was aborted because states could not be created successfully. Error: Gateway Time-out