500699 - deploy puppet client on production machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Comment 1

[[e-mail] (:bhearsum)]

Got through moz2-linux-slave01 -> slave12 today. One of them hit an OOM problem, I'm going to let things run overnight before I continue deployment. We've also got bug 501222 filed to give the linux slaves more ram. Hopefully this won't block the rest of the puppet deployment.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

We hit this on a couple of builds on moz2-win32-slave09 (m-c unit). As an experiment I rebooted that to see what happened to the memory usage.

Before:
             total       used       free     shared    buffers     cached
Mem:           757        326        430          0          2        276
-/+ buffers/cache:         46        710
Swap:          509         62        447

After:
             total       used       free     shared    buffers     cached
Mem:           757        296        460          0         15        208
-/+ buffers/cache:         72        684
Swap:          509          0        509

So using about 90M less in RAM and swap. Uptime was 45 days.

Comment 3

[[e-mail] (:bhearsum)]

Unfortunately we had 5 more failures due to memory exhaustion overnight. I've turned off the puppet service for now. I'm going to continue deployment today but I will not be starting the puppet daemon, merely doing the installations.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #2)
> We hit this on a couple of builds on moz2-win32-slave09 (m-c unit). As an
> experiment I rebooted that to see what happened to the memory usage.

I'm so used to windows being a pain, but really mean moz2-linux-slave09 here.

Comment 5

[[e-mail] (:bhearsum)]

I deployed puppet and did an initial sync up on the rest of the moz2-* slaves - I have not enabled the service yet.

Comment 6

[[e-mail] (:bhearsum)]

I got puppet deployed, albeit disabled, on the try-linux-* slaves today. Two things left: deploy on the ref platform, and enable the service. The latter is blocked on bug 501222.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

bhearsum: can you turn on the puppet service on moz2-linux-slave17, as it now has 2gb RAM. I'm curious if that RAM upgrade fixed the memory corruption as hoped.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to comment #7)
> bhearsum: can you turn on the puppet service on moz2-linux-slave17, as it now
> has 2gb RAM. I'm curious if that RAM upgrade fixed the memory corruption as
> hoped.

Done.

Comment 9

[[e-mail] (:bhearsum)]

Got the reference platform done today. Deployment is done now, the only thing left is turning on the service once bug 501222 is done.

Comment 10

[[e-mail] (:bhearsum)]

Puppet has been enabled on all of the try-* and moz2-* linux machines. We're all done here, finally!

Comment 11

[[e-mail] (:bhearsum)]

catlee found a major hole in this deployment today: /etc/sysconfig/puppet was never updated to point to the puppet service. This meant that puppet was running every 30min, but only using the cached manifests from the last time puppetd was run manually. We went around and fixed all the slaves, including the ref platform so it's fixed now.

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #11)
> catlee found a major hole in this deployment today: /etc/sysconfig/puppet was
> never updated to point to the puppet service. This meant that puppet was
> running every 30min, but only using the cached manifests from the last time
> puppetd was run manually. We went around and fixed all the slaves, including
> the ref platform so it's fixed now.

Yikes. (Nice catch Catlee!). So we've fixed it this time, but is there any way to make sure this doesnt hit us again in the future?

Comment 13

[[e-mail] (:bhearsum)]

This was something that was missed during the initial setup. I fixed the ref platform, too, so new slaves will be fine. Unless something or someone changes /etc/sysconfig/puppet there's no way for this to re-occur.