1445697 - Workers getting 403's renewing statsum/signalfx creds

Status: RESOLVED FIXED

(Taskcluster :: Workers, enhancement, P1)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We're seeing lots of these, causing lots of alerts about excessive 403's.

docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 for example:

https://papertrailapp.com/systems/1687026791/events?focus=910560469327179819

startup:

Mar 14 12:49:13 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="1496" x-info="http://www.rsyslog.com"] start 
...
Mar 14 12:49:41 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 docker-worker: {"type":"configure","source":"host/aws","url":"http://169.254.169.254/latest"} 

takes a task:

Mar 14 12:49:44 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 docker-worker: {"type":"task start","source":"top","provisionerId":"aws-provisioner-v1","workerId":"i-066bcbd52ede5fb37","workerGroup":"us-west-1","workerType":"gecko-t-linux-large","workerNodeType":"m3.large","taskId":"Re3EkNjpT0Ocb8qvdsEaeg","runId":0} 
...
Mar 14 13:17:54 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 docker-worker: {"type":"task resolved","source":"top","provisionerId":"aws-provisioner-v1","workerId":"i-066bcbd52ede5fb37","workerGroup":"us-west-1","workerType":"gecko-t-linux-large","workerNodeType":"m3.large","taskId":"Re3EkNjpT0Ocb8qvdsEaeg","runId":0,"taskState":"completed"} 

takes a task:

Mar 14 13:20:52 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 docker-worker: {"type":"task start","source":"top","provisionerId":"aws-provisioner-v1","workerId":"i-066bcbd52ede5fb37","workerGroup":"us-west-1","workerType":"gecko-t-linux-large","workerNodeType":"m3.large","taskId":"GkElnORJT7ul6Vc4ilkjww","runId":1} 

OOM:

Mar 14 13:27:12 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 docker-worker: Uncaught Exception! Attempting to report to Sentry and crash.
Mar 14 13:27:13 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 docker-worker: Error: spawn ENOMEM 

...


Mar 14 13:29:14 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 kernel: [ 2409.357896] Out of memory: Kill process 11672 (firefox) score 869 or sacrifice child
Mar 14 13:29:14 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 kernel: [ 2409.362560] Killed process 11725 (Web Content) total-vm:1649800kB, anon-rss:42604kB, file-rss:7268kB
Mar 14 13:29:15 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 docker-worker: Failed to report error to Sentry after timeout!

and here's the issue, I think:

Mar 14 13:29:15 docker-worker.aws-provisioner.us-west-1c.ami-cc909aac.m3-large.i-066bcbd52ede5fb37 kernel: [ 2410.534578] init: docker-worker main process ended, respawning 

So that new process can't get any credentials, since the secret has been deleted.

Should we just shut down in this case, instead?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Wander, it seems like hte underlying issue is that docker-worker restarted on the host.  I think we've decided to just live with that problem?  If so, let's WONTFIX..

Comment 2

[Wander Lairson Costa]

We now shut down on this error [1].

[1] https://github.com/taskcluster/docker-worker/blob/master/src/lib/host/aws.js#L136