1503164 - I'm unable to terminate or quarantine workers through tools.taskcluster.net

Status: RESOLVED WORKSFORME

(Taskcluster :: UI, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I tried to quarantine a worker earlier today and hit a weird error message when I submitted the request.

Error was:
Error AuthenticationFailed
ext.certificate.expiry < now

This sends off several PUT requests that result in 401 status errors. Trying the same with the "terminate" button results in a bunch of DELETE requests that also have a failing status code.

At the time, I had been signed in to taskcluster for quite a while, and my username up in the corner of the screen showed "email/[e-mail]".

Thinking maybe that I just needed to sign out and back in to refresh something with Auth0, I did that. 

The name up in the corner changed to "Wesley Kocher". Trying the "quarantine" and "terminate" buttons with my newly-refreshed credentials, the errors changed to:
Error InternalServerError
Internal Server Error, incidentId: 8a033db4-8f42-451a-b229-81800084f2d6




That first error could really be more helpful. If I need to re-sign in to refresh credentials, it should just say that.

The second error is more confusing. Maybe it's because I picked a worker that last had a job 20 hours ago, so maybe it was already terminated or something? I don't want to test that theory by canceling a random worker that's actively doing work, though.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Signing in with your email is not enough to get scopes for this sort of thing - definitely sign in with your LDAP credentials.  I'm surprised Auth0 let you sign in with your email, to be honest (it should require that you use LDAP if an LDAP with that email exists).  Maybe you had "manually" logged in with a temporary credential ages ago?  Or maybe your laptop time is skewed?  Everything about that first part of comment 0 is 110% strange.

Anyway, I can't find that incident ID.  Can you reproduce and provide the HTTP request (from the network devtools pane) and timestamp?

Comment 2

[Pete Moore [:pmoore][:pete]]

FWIW I also independently searched for that incidentId this morning in the papertrail logs, and couldn't find it. I've not had that before, it is a little concerning. If it happens again (that an incidentId can't be found in log history) we can create a bug for that as a separate issue. Weird.

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I've been unable to reproduce this (haven't really been trying and haven't had a need to attempt to cancel anything lately). This can probably be closed worksforme unless something obvious is wrong for someone else.