1020756 - mysql2, postgres puppet modules: wrap popen3 calls in "if binary exists"

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[:Atoll]

For some unknown reason, the popen3 calls in the mysql_* and postgres_*.rb custom facts are breaking on ftp5.dmz.scl3, which has neither mysql nor psql binaries to run.

This doesn't affect the facts using Facter's exec routine, and these are the only facts we have that use popen3 directly.

Wrapping the popen3 calls in "if exists in PATH" or "if exists at path", depending on the method used, should fix this, as I'm guessing there's some missing popen3 code for handling "command not found" properly.

Comment 1

[:Atoll]

mysql2 commit r88590, postgres commit r88591.

The mysql2 commit wraps each call to hard-coded '/usr/bin/mysql' in "if File.exists?('/usr/bin/mysql')", so that it only runs the popen3 if the binary exists. This fixes the mysql facts on ftp5, and has no effect on the mysql facts shown on other servers where MySQL is actually running.

The postgres commit wraps each call to PATH-dependent 'psql' in "unless Facter::Util::Resolution.which('psql').nil?", so that it only runs the popen3 if the binary is found in the PATH. This fixes the postgres facts similarly, and has no effect on postgres servers as well.

Comment 2

[:Atoll]

:rtucker and :dustin, I've never seen popen3 used in a custom fact, and I'm guessing this "odd" timeout issue backtrace stems from some interaction between popen3, Facter, and/or Ruby.

The above commits fix everything, so this is merely an advisory to be aware in case you come across this sort of thing in the future.

The actual error returned is:

Failed to apply catalog: Could not retrieve local facts: No child processes

And if you run 'facter --puppet --debug --trace', it emits a backtrace at the popen3 line inside the specific custom fact having issues, so it's easy enough to diagnose once it happens.

Comment 3

[Rob Tucker [:rtucker]]

The reason for the popen3(In reply to Richard Soderberg [:atoll] from comment #2)
> :rtucker and :dustin, I've never seen popen3 used in a custom fact, and I'm
> guessing this "odd" timeout issue backtrace stems from some interaction
> between popen3, Facter, and/or Ruby.
> 
> The above commits fix everything, so this is merely an advisory to be aware
> in case you come across this sort of thing in the future.
> 
> The actual error returned is:
> 
> Failed to apply catalog: Could not retrieve local facts: No child processes
> 
> And if you run 'facter --puppet --debug --trace', it emits a backtrace at
> the popen3 line inside the specific custom fact having issues, so it's easy
> enough to diagnose once it happens.

Awesome, thanks!

Comment 4

[:Atoll]

> ::popen3 can also take a block which will receive stdin, stdout and stderr as parameters. This ensures stdin, stdout and stderr are closed once the block exits. Example:

We need to rewrite these in block form (popen3() { |in,out,err| ... }) so the FDs are autoclosed, as this may be why things are hanging/crashing. I've already tested the block form and it works fine, so it should be a noop in any case.

Comment 5

[:Atoll]

sysadmins r88680 converts the popen3 calls to { |i,o,e| } style, moving the Facter setcode call *outside* the popen3 block, as otherwise Facter crashes internally (hooray).

Confirmed no changes to facts on postgres and mysql servers.