1080645 - puppet doesn't run from cron on Trusty

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

On proxxy1, puppet isn't running.  I tried restarting crond, to no effect.

Comment 1

[:kanban-engops]

A Pivotal Tracker story has been created for this Bug: https://www.pivotaltracker.com/story/show/80405514

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

In bug 1057036 I hypothesized that this might be related to bug 1033112, but a restart of crond would have fixed that, and didn't.  I also don't see any logging from cron, as I did in bug 1033112.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Cron isn't even *looking* at the puppetcheck.cron file, but is looking at `anacron` in the same directory (this is after touching puppetcheck.cron)

stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0
stat("crontabs", {st_mode=S_IFDIR|S_ISVTX|0730, st_size=4096, ...}) = 0
stat("/etc/crontab", {st_mode=S_IFREG|0644, st_size=722, ...}) = 0
stat("/etc/cron.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/etc/cron.d/anacron", {st_mode=S_IFREG|0644, st_size=188, ...}) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {0x4031d0, [CHLD], SA_RESTORER|SA_RESTART, 0x7f3c1832eff0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({60, 0}, 0x7fff03346270)      = 0

Perms are identical:

proxxy1 ~ # ls -al /etc/cron.d
total 28
drwxr-xr-x   2 root root  4096 Oct 10 06:33 .
drwxr-xr-x 125 root root 12288 Oct 10 06:33 ..
-rw-r--r--   1 root root   188 Oct 10 06:38 anacron
-rw-r--r--   1 root root   102 Feb  8  2013 .placeholder
-rw-r--r--   1 root root   247 Oct 10 06:39 puppetcheck.cron

If I touch /etc/cron.d:

proxxy1 ~ # touch /etc/cron.d
proxxy1 ~ # 0x7fff03346270)      = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0
stat("crontabs", {st_mode=S_IFDIR|S_ISVTX|0730, st_size=4096, ...}) = 0
stat("/etc/crontab", {st_mode=S_IFREG|0644, st_size=722, ...}) = 0
stat("/etc/cron.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/etc/crontab", {st_mode=S_IFREG|0644, st_size=722, ...}) = 0
open("/etc/crontab", O_RDONLY)          = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=722, ...}) = 0
close(4)                                = 0
openat(AT_FDCWD, "/etc/cron.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
getdents(4, /* 5 entries */, 32768)     = 152
lstat("/etc/cron.d/anacron", {st_mode=S_IFREG|0644, st_size=188, ...}) = 0
open("/etc/cron.d/anacron", O_RDONLY)   = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=188, ...}) = 0
close(5)                                = 0
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0
openat(AT_FDCWD, "crontabs", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
getdents(4, /* 3 entries */, 32768)     = 72
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 5
lseek(5, 0, SEEK_CUR)                   = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=5474, ...}) = 0
mmap(NULL, 5474, PROT_READ, MAP_SHARED, 5, 0) = 0x7f3c18d0b000
lseek(5, 5474, SEEK_SET)                = 5474
munmap(0x7f3c18d0b000, 5474)            = 0
close(5)                                = 0
open("crontabs/root", O_RDONLY|O_NOFOLLOW) = 5
fstat(5, {st_mode=S_IFREG|0600, st_size=597, ...}) = 0
close(5)                                = 0
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {0x4031d0, [CHLD], SA_RESTORER|SA_RESTART, 0x7f3c1832eff0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({60, 0}, 0x7fff03346270)      = 0

so it *reads* the directory but decides to ignore puppetcheck.cron based solely on its dirent.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

..but remove the '.cron' suffix and it's ALL OVER IT:

0x7fff03346270)      = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2819, ...}) = 0
stat("crontabs", {st_mode=S_IFDIR|S_ISVTX|0730, st_size=4096, ...}) = 0
stat("/etc/crontab", {st_mode=S_IFREG|0644, st_size=722, ...}) = 0
stat("/etc/cron.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/etc/crontab", {st_mode=S_IFREG|0644, st_size=722, ...}) = 0
open("/etc/crontab", O_RDONLY)          = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=722, ...}) = 0
close(4)                                = 0
openat(AT_FDCWD, "/etc/cron.d", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
getdents(4, /* 5 entries */, 32768)     = 144
lstat("/etc/cron.d/puppetcheck", {st_mode=S_IFREG|0644, st_size=247, ...}) = 0
open("/etc/cron.d/puppetcheck", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=247, ...}) = 0
fcntl(5, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(5, {st_mode=S_IFREG|0644, st_size=247, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c18d0c000
lseek(5, 0, SEEK_CUR)                   = 0
read(5, "MAILTO=\"[e-mail]"..., 4096) = 247
lseek(5, 247, SEEK_SET)                 = 247

That's a little ridiculous, but it IS documented in the manpage:

> Files in this directory have to be owned by root, do not need to be
> executable (they are configuration files, just like /etc/crontab) and
> must conform to the same naming convention as used by run-parts(8):
> they  must  consist solely  of upper- and lower-case letters, digits,
> underscores, and hyphens. This means that they cannot contain any dots.
> If the -l option is specified to cron (this option can be setup through
> /etc/default/cron, see below), then they must conform to the LSB
> namespace specification, exactly as in the --lsbsysinit option in
> run-parts.

We should probably fix this name everywhere, not just on Trusty.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1080645.patch

Comment 7

[Jake Watkins [:dividehex]]

Comment on attachment 8503158 [details] [diff] [review]
bug1080645.patch

At least it was documented.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, and because /bin/sh is dash, it doesn't understand 'source', so we should use '.' instead.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1080645-p1.patch

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8504054 [details] [diff] [review]
bug1080645-p1.patch

Review of attachment 8504054 [details] [diff] [review]:
-----------------------------------------------------------------

This makes total sense. Haven't known we use dash there.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

For reference the patch landed as http://hg.mozilla.org/build/puppet/rev/690b0c803b46