1461913 - Don't continue running generic-worker if it returned with exit code 69

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Pete Moore [:pmoore][:pete]]

If generic-worker detects it is in a bad state (e.g. can't run tasks due to i/o problems or corrupt data), it returns with exit code 69.

If this happens, we should disable the worker, and ideally raise a bug in bugzilla for someone to look into it.

Recently we hit an issue with t-yosemite-r7-449 boot-looping.

I've created this bug so if in future a worker gets into a bad state, it doesn't eat through tasks or repeatedly crash, sending alerts.

The generic-worker --help command explains what the 64-72 exit codes mean:

>     69     Worker panic - either a worker bug, or the environment is not suitable for running
>            a task, e.g. a file cannot be written to the file system, or something else did
>            not work that was required in order to execute a task. See config setting
>            shutdownMachineOnInternalError.

Comment 1

[Dragos Crisan [:dragrom]]

Created RelOps bugzilla user for filing bugs via API. Added all information to passwords/bugzilla.gpg file stored in ssh://[e-mail]/relops/gpg.git repository

dhouse: Can you please make changes to your code and remove the existing api key?

Comment 2

[:dhouse]

(In reply to Dragos Crisan [:dragrom] from comment #1)
> Created RelOps bugzilla user for filing bugs via API. Added all information
> to passwords/bugzilla.gpg file stored in
> ssh://[e-mail]/relops/gpg.git repository
> 
> dhouse: Can you please make changes to your code and remove the existing api
> key?

Thank you for setting the user up. I changed roller prod to use the relops@ bugzilla user's apikey from the gpg repo. So roller and the quarantineworker will be using the same apikey; I don't think bugzilla provides different permissions for different apikeys for the same user, and so I think it is fine to not make separate keys.

The key is stored, for roller prod to use, in releng-puppet's hiera as roller_bugzilla_api_key!prod. Roller dev is set up with a separate apikey for the dev bugzilla instance.

Comment 3

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for build-puppet

A Pull Request from Dragos for implementing this in build-puppet repo.

Comment 4

[Dragos Crisan [:dragrom]]

Attachment: Don't continue running generic-worker if it returned with exit code 69

Check the status code returned by generic worker
If status code is 69, quarantine worker and create the bug
If the worker will be back to normal, then remove the worker from the quarantine and close the bug

Comment 5

[Dragos Crisan [:dragrom]]

To reproduce the exit code 69, just change the ownership for /Users/cltbld/generic-worker.openpgp.key file