1186542 - issues with b-2008, try-2008 userdata in build-cloud-tools

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Attachment: one way to improve things possibly

I've found a number of issues with the windows userdata script that i'd like some  discussion on before I attempt a fix (in case, I'm completely wrong).

- When the golden instance spins up, it already contains a puppet service and scheduled task to run the puppet agent. The service is configured to start automatically and it does. The userdata script attempts to resolve this with an `sc delete puppet` call, but this only partially succeeds. What actually happens is that the service receives a delete instruction, which conflicts with its current running status. To resolve this, it postpones deletion until it is no longer busy. This running service instance is what has been giving us our limited success puppet runs.

- The userdata script had some unterminated quoted strings and unterminated opening parentheses. These were causing commands within the script to fail or to run unpredictably. I'm not sure which because the output from the script goes to a non-interactive process and isn't logged. In any case, the timing loops and gotos succeeded in keeping the instance alive long enough for the separate puppet service process to sometimes complete. Perhaps the scripts own puppet agent command was also running. Perhaps the scheduled task was making an effort too. It's hard to say. But the picture I get is one where we don't really know which puppet processes are working and which aren't.

- the puppet agent call from the userdata script was wrapped in a call to `cmd /c`. I'm not sure if there was a good reason for this or if it just snuck in with a copy/paste from elsewhere. It has the effect of spawning a new process for the puppet agent run and making all those ugly wait loops and gotos necessary. Without the `cmd /c` wrapper, the script will wait for the puppet call to complete before moving on to the next step.

- according to http://docs.puppetlabs.com/puppet/4.2/reference/services_agent_windows.html#logging, "When running in the foreground with the --verbose, --debug, or --test options, Puppet agent logs directly to the terminal." For us this effectively logs to /dev/null. I believe the event log puppet messages can be attributed to the running-but-waiting-to-terminate puppet service. This can be fixed by adding a `--logdest <FILE>` argument to the puppet agent call, but it's missing in userdata.

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

I'd actually prefer to replace the cmd script with a powershell one like this: https://gist.github.com/grenade/1e7bb5545ddb93368bb9#file-try-2008-userdata-ps1 but I'm currently having problems getting build-cloud-tools to handle scripts that have a <powershell> tag rather than the cmd <script> tag.

Comment 2

[Mark Cornmesser [:markco]]

(In reply to Rob Thijssen (:grenade - GMT) from comment #0)
> Created attachment 8637350 [details] [diff] [review]
> one way to improve things possibly
> 
> I've found a number of issues with the windows userdata script that i'd like
> some  discussion on before I attempt a fix (in case, I'm completely wrong).
> 
> - When the golden instance spins up, it already contains a puppet service
> and scheduled task to run the puppet agent. The service is configured to
> start automatically and it does. The userdata script attempts to resolve
> this with an `sc delete puppet` call, but this only partially succeeds. What
> actually happens is that the service receives a delete instruction, which
> conflicts with its current running status. To resolve this, it postpones
> deletion until it is no longer busy. This running service instance is what
> has been giving us our limited success puppet runs.

That's strange and should not be the case. The schedule task to run Puppet should not be there. Puppet should be installed, the service should be running by default, but it should not be authenticated or have a schedule task in place already. Do we know what the starting ami ID is?  

We want to control the puppet run and have it only run once in AWS. The service should be killed as soon as possible, so we don't have the conflicting issues and so it doesn't pick up changes during a build. The schedule task should not be there because it is managed by Puppet. Which is why i think the wrong ami is being used. 

As far as the userdata, maybe something happen during a check in. What i have been using locally has been working as I expected it to.

Comment 3

[Mark Cornmesser [:markco]]

I was thinking about this earlier. By no means are we stuck to the current userdata. If you have a cleaner way of doing it then I am all for it. 

What we are looking for:

Delete Puppet service (until we recapture the base mdt image with it deleted). 

Puppet certification. Which requires the setx commands, so it matches the dns entries, and running the pupeptize vbs script.

Puppet run until success. Success requiring and exit code of 2 or 0. In order to get actual exit codes we need to use the flag "--detailed-exitcodes". Overall we should probably use the same flags that are being used here: http://hg.mozilla.org/build/puppet/file/6b321c7f848c/modules/puppet/templates/puppet-atboot-runpuppet.rb.erb 

And lastly delete the run puppet task. This could be managed through Puppet. I have left here as way to do a quick check if Puppet completed successfully and that the userdata executed completed.

Comment 4

[Amy Rich [:arr] [:arich]]

One thing that has not been explicitly mentioned as a requirement... when it comes to puppetizing the golden instance, we need to make sure that the cloud-tools scripts do not shut it down and snapshot it until the puppet run is successful. I believe this is why that logic was moved to the userdata in the first place.

Comment 5

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Fixed in PR: https://github.com/mozilla/build-cloud-tools/pull/93