1454911 - Run-Puppet :: puppet agent summary contains error

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Adrian Pop]

Today, while monitoring the channel #buildduty, this notification appeared :

[sns alert] Apr 18 10:16:11 y-2008-ec2-golden.try.releng.use1.mozilla.com Userdata: Run-Puppet :: puppet agent summary contains error#015

Can you please help with this issue or point me to someone who knows how this issue can be solved ?

Comment 1

[Adrian Pop]

I have tried to log on the server via ssh, but I have received the following error : 

Unable to negotiate with UNKNOWN port 65535: no matching cipher found. Their offer: 3des-cbc,blowfish-cbc,cast128-cbc,idea-cbc

Comment 2

[Adrian Pop]

this seems to be the commit that generates the issue : https://hg.mozilla.org/build/puppet/rev/d3a68c812267

Comment 3

[Adrian Pop]

the golden ami should be recreated after the problem with the puppet has been resolved :

Wed 13:41:06 UTC [7891] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 15 crit, 0 warn out of 15 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)

Comment 4

[Jake Watkins [:dividehex]]

This looks like it is affecting all 

Off the top of my head there are 2 things this could be:

1: golden amis instances include puppet::periodic that causes them to also include the puppet::renew which in turn fails because it doesn't support windows.  My assumption was that golden images only run puppet during creations and therefore do not include puppet::periodic

2: Instance data json files contain puppetmasters to puppetize against.  They were changed yesterday to only point to MDC1/2, USE1, and USW2.  SCL3 masters were removed completely.  This means golden images tried to puppetize elsewhere but may be blocked by flows.

Comment 5

[Jake Watkins [:dividehex]]

This is the commit that changes the puppetmasters available to AWS instances to puppetized against

https://github.com/mozilla-releng/build-cloud-tools/commit/0a5449b2a03640a77b4022b50a58cbe0a68a4c82

Comment 6

[Jake Watkins [:dividehex]]

Opps, didn't finish a thought.

This looks like it is affecting all WINDOWS golden ec2 instances.

Comment 7

[Jake Watkins [:dividehex]]

This facter module should explicitly define its confined OSes that it supports

2018-04-18 01:45:42 -0700 /File[C:/ProgramData/PuppetLabs/puppet/var/lib/facter/puppet_agent_cert.rb]/ensure (notice): defined content as '{md5}34ac2cb59cbf1af8f0213b8df5bc879d'

2018-04-18 01:45:47 -0700 Puppet (err): Could not retrieve local facts: uninitialized constant Facter::Core
2018-04-18 01:45:47 -0700 Puppet (err): Failed to apply catalog: Could not retrieve local facts: uninitialized constant Facter::Core

Comment 8

[Jake Watkins [:dividehex]]

Attachment: Limit cert renew manifest and facter to Linux and Darwin only

This limits puppet::renew_cert and puppet_agent_cert.rb facter module from running on windows

Comment 9

[Jake Watkins [:dividehex]]

Comment on attachment 8968979 [details] [diff] [review]
Limit cert renew manifest and facter to Linux and Darwin only

https://hg.mozilla.org/build/puppet/rev/c1f4dc76da6979282dbd0fc631d736aff9a94489
https://hg.mozilla.org/build/puppet/rev/0309058909d8525b7020ec61e183b9e1e7806887

Comment 10

[Jake Watkins [:dividehex]]

(In reply to Jake Watkins [:dividehex] from comment #9)
> Comment on attachment 8968979 [details] [diff] [review]
> Limit cert renew manifest and facter to Linux and Darwin only
> 
> https://hg.mozilla.org/build/puppet/rev/
> c1f4dc76da6979282dbd0fc631d736aff9a94489
> https://hg.mozilla.org/build/puppet/rev/
> 0309058909d8525b7020ec61e183b9e1e7806887

This changeset has propagated to all the puppetmasters by now.  It should be safe to kill and restart the golden image generation for the windows hosts that failed earlier.

Comment 11

[Zsolt Fay [:zfay]]

y-2008-ec2-golden.try.releng.use1.mozilla.com Userdata: Run-Puppet :: puppet agent summary contains error#015.
^^Still coming in even after I terminated y-2008-ec2-golden and the daily cron re-created it. 

Any updates on the #015 error?

Comment 12

[Jake Watkins [:dividehex]]

(In reply to Zsolt Fay [:zsoltfay] from comment #11)
> y-2008-ec2-golden.try.releng.use1.mozilla.com Userdata: Run-Puppet :: puppet
> agent summary contains error#015.
> ^^Still coming in even after I terminated y-2008-ec2-golden and the daily
> cron re-created it. 
> 
> Any updates on the #015 error?

The patch I applied fixed the bustage caused by the puppet_agent_cert.rb.  After that it looks like it was still suffering from problems caused by another patch (https://hg.mozilla.org/build/puppet/rev/07e4d452e404).  I pinged bhearsum in #buildduty and he push more patches that should have fixed the issue.  As of right now, I've seen  y-2008-ec2-golden.try.releng.use1.mozilla.com successfully complete a puppet run.

Therefore, I believe this issue is fixed.

Comment 13

[Zsolt Fay [:zfay]]

I'll mark this as fixed then. Will re-open it if y-2008-ec2-golden misbehaves.