1187937 - docker-worker fails to initialize with "Gateway Time-out"

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Rail Aliiev [:rail]]

Similar to Bug #1187934, but different signature.

Sounds like it's trying to initialize the link, but fails. Probably should be an infra error, so TC reschedules the task.

[taskcluster] taskId: IiFbGm0zS8qWPAlM44d7oA, workerId: i-46838995

[taskcluster] Error: Task was aborted because states could not be created successfully. Error: Error: Gateway Time-out
[taskcluster] Unsuccessful task run with exit code: -1 completed in 923.868 seconds

Comment 1

[Rail Aliiev [:rail]]

This is #1 failure for funsize now. Tens a day, see https://treeherder.allizom.org/#/jobs?repo=mozilla-aurora&exclusion_profile=false&filter-searchStr=funsize

Any suggestions regarding what and how I should tackle to make this retry better?

Comment 2

[Greg Arndt [:garndt]]

I'm going to ni? myself on this so I can check it out for you.

Comment 3

[Rail Aliiev [:rail]]

Now we starting using these for releases! \o/

Comment 4

[Rail Aliiev [:rail]]

Greg, what do you think if we bump default 60s timeout for the following commands to something like 120?

https://github.com/taskcluster/taskcluster-vpn-proxy/compare/master...rail:gateway_timeout?expand=1

Comment 5

[Greg Arndt [:garndt]]

I'm definitely up for it.  If you want to open up the PR, I could get it merged and a new proxy image deployed and used by those workers

Comment 6

[Rail Aliiev [:rail]]

Attachment: Increase timeouts

(In reply to Greg Arndt [:garndt] from comment #5)
> I'm definitely up for it.  If you want to open up the PR, I could get it
> merged and a new proxy image deployed and used by those workers

Let's do it! :)

Comment 7

[Greg Arndt [:garndt]]

This has been merged, but note the following steps I need to complete and based on some stuff going on and being late in the week might take me a little time:

1. Build vpn proxy image
2. Update docker-worker to reference that new version
3. deploy new docker-worker to those workertypes

Comment 8

[Rail Aliiev [:rail]]

There is no rush. Feel free to postpone this to next week.

Comment 9

[Greg Arndt [:garndt]]

You're the best, thank you!

Comment 10

[Greg Arndt [:garndt]]

I'll assign this to myself for the next pieces that need to happen, and I'll update this bug.

Comment 11

[Greg Arndt [:garndt]]

New amis should be rolling out, please let me know if you see any issues.

Comment 12

[Rail Aliiev [:rail]]

woot! Thanks a lot!

Comment 13

[Rail Aliiev [:rail]]

Seen this once today. I wonder if we should schedule a rerun in this case...

Comment 14

[Rail Aliiev [:rail]]

Doesn't look that increasing the timeout resolves the issue. https://tools.taskcluster.net/task-inspector/#KjPszsj5RoezXmL4yVgc1A/0

Maybe we should look at abortRun() and just rerun until we reach `retries`?

Comment 15

[Greg Arndt [:garndt]]

You should be able to set the retry number when submitting the graph of these tasks and if the task fails, it will retry up to that number of times.  Maybe that's enough to work around this? I believe we are moving away from requiring vpn at all.

Comment 16

[Rail Aliiev [:rail]]

I think it's already set, "retries": 5, see https://queue.taskcluster.net/v1/task/KjPszsj5RoezXmL4yVgc1A

Comment 17

[Greg Arndt [:garndt]]

That's how many times a task will retry if there was an exception reported (worker-shutdown for instance).  These tasks right now are marked as "failed" and the only way they get retried is by specifying "reruns" when submitting the task graph.

This is listed in the request payload for a task node here: http://docs.taskcluster.net/scheduler/api-docs/#createTaskGraph

Comment 18

[Rail Aliiev [:rail]]

Oh, sorry, mixed reruns and retries. Let's close this bug, I'll address this by specifying reruns.

Comment 19

[Rail Aliiev [:rail]]

Yup, setting "reruns" solves the issue: https://tools.taskcluster.net/task-inspector/#Fd9OCbp_QjerTJchTBR1CA/0

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Great to hear that we were able to workaround this issue. But shouldn't we still investigate what's actually broken here? It could still be that reruns are broken because of that problem.

Comment 21

[Rail Aliiev [:rail]]

Balrog VPN proxy will be dead soon, we are going to use balrog workers instead. IMO it's not worth to invest more into fixing the current approach.