991093 - spot instances trying to use revoked puppet certs

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

At least the following machines are failing to come up because of certificate revocation:
try-linux64-spot-137.try.releng.use1.mozilla.com
try-linux64-spot-036.try.releng.use1.mozilla.com
try-linux64-spot-183.try.releng.use1.mozilla.com

Comment 1

[Chris AtLee [:catlee]]

Attachment: list of revoked certs

There are 300 revoked certs. I've moved them out of the way into ~/revoked on aws-manager.

Comment 2

[Chris AtLee [:catlee]]

I found revoked certs with this command (in /builds/aws_manager/secrets/cached_certs)

for f in *; do if ( ! echo "" | openssl s_client -host releng-puppet2.srv.releng.scl3.mozilla.com -port 8140 -quiet -no_ign_eof -cert $f > ~/$f.log 2>&1 ); then echo $f bad; else rm ~/$f.log; fi; done

then I double checked that the log files indicated the revocation as opposed to some other error

Comment 3

[Rail Aliiev [:rail]]

Boo, bug 986477 will help here a lot.

Comment 4

[Rail Aliiev [:rail]]

No more puppet on spot instances