1152371 - Investigate Puppet run errors for 2008 in AWS

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Mark Cornmesser [:markco]]

The following errors occur on the AWS machine and has only on an AWS machine. I have not been able to recreate in the datacenter.

Tue Apr 07 22:20:36 -0700 2015 Puppet (err): Failed to set owner to 'S-1-5-21-2428560543-3347429064-4102443262-500': Failed to add access control entry: No more memory is available for security information updates. 

Tue Apr 07 22:20:36 -0700 2015 /Stage[main]/Packages::Mozilla::Mozilla_build/File[C:\mozilla-build/hg/hgrc.d/Paths.rc]/content (err): change from {md5}b63f14ff0fd584287e350af280832f89 to {md5}aaf28662d4652febd343a79083f183d9 failed: Failed to set owner to 'S-1-5-21-2428560543-3347429064-4102443262-500': Failed to add access control entry: No more memory is available for security information updates. 

Tue Apr 07 22:20:36 -0700 2015 Puppet (err): Failed to set owner to 'S-1-5-21-2428560543-3347429064-4102443262-500': Failed to add access control entry: No more memory is available for security information updates. 

Tue Apr 07 22:20:36 -0700 2015 /Stage[main]/Packages::Mozilla::Mozilla_build/File[C:\mozilla-build/hg/hgrc.d/Paths.rc]/owner (err): change from BUILTIN\Administrators to B-2008-EC2-0005\root failed: Failed to set owner to 'S-1-5-21-2428560543-3347429064-4102443262-500': Failed to add access control entry: No more memory is available for security information updates.

Comment 1

[Mark Cornmesser [:markco]]

When I kick off  a run manually I get this new set of errors:

Wed Apr 08 14:31:31 -0700 2015 Puppet (err): Could not set 'file' on ensure: No such file or directory - C:/etc/MozBld_ver_check.bat20150408-896-1u4id30.lock at 18:/etc/puppet/production/modules/packages/manifests/mozilla/mozilla_build.pp 

Wed Apr 08 14:31:31 -0700 2015 Puppet (err): Could not set 'file' on ensure: No such file or directory - C:/etc/MozBld_ver_check.bat20150408-896-1u4id30.lock at 18:/etc/puppet/production/modules/packages/manifests/mozilla/mozilla_build.pp 
Wrapped exception: 
No such file or directory - C:/etc/MozBld_ver_check.bat20150408-896-1u4id30.lock 

Wed Apr 08 14:31:31 -0700 2015 /Stage[main]/Packages::Mozilla::Mozilla_build/File[C:/etc/MozBld_ver_check.bat]/ensure (err): change from absent to file failed: Could not set 'file' on ensure: No such file or directory - C:/etc/MozBld_ver_check.bat20150408-896-1u4id30.lock at 18:/etc/puppet/production/modules/packages/manifests/mozilla/mozilla_build.pp 

Wed Apr 08 14:33:50 -0700 2015 Puppet (err): Could not find command 'C:\mozilla-build\python\python2.7.exe' 
Wed Apr 08 14:33:50 -0700 2015 /Stage[main]/Instance_metadata/Exec[get_instance_metadata]/returns (err): change from notrun to 0 failed: Could not find command 'C:\mozilla-build\python\python2.7.exe' 



Which is odd because these errors are not occurring in the datacenter.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

That looks like an ordering issue -- trying to create a file in c:\etc before c:\etc is created.  Try adding 'include dirs::etc'?

Comment 3

[Mark Cornmesser [:markco]]

The ACl errors were being caused by a difference int he facter fdqn and the actual machine name. The disparage was just within the testing environment.

Also note after clearing up the ACL errors I started running into disk space errors. Puppet was not completing because of a lack of space.It seems we need a minimum of +/- 40 GB. for the Puppet run to complete. 

The errors that appear to be an ordering issue only occurs when the Puppet run is manually kicked off. I was able to recreate with in the data center by using the cloud task sequence to install and then manually kicking off the Puppet run. However, when the puppet run is triggered by EC2 user data it does not throw the same errors.

Comment 4

[Mark Cornmesser [:markco]]

Attachment: bug1152371.patch

I don't understand how puppet run completed when passed by the user data for AWS. Which is a bit concerning, but this patch clears the errors for when puppet is manually kicked off.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8592357 [details] [diff] [review]
bug1152371.patch

Review of attachment 8592357 [details] [diff] [review]:
-----------------------------------------------------------------

Ordering bugs can come and go strangely.  This looks like the right fix.