1095160 - runner shouldn't block other startup scripts

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

Attachment: swap.diff

LVM broke our ability to use swap. :/

Comment 1

[Rail Aliiev [:rail]]

Comment on attachment 8518502 [details] [diff] [review]
swap.diff

Actually it doesn't help. We should fall back to /swapfile

Comment 2

[Rail Aliiev [:rail]]

it turns out that runner blocks other init scripts to start:

lrwxrwxrwx 1 root root 16 Nov  6 01:20 S98runner -> ../init.d/runner
lrwxrwxrwx 1 root root 18 Nov  6 01:22 S99add_swap -> ../init.d/add_swap
lrwxrwxrwx 1 root root 18 Nov  6 01:24 S99collectd -> ../init.d/collectd
lrwxrwxrwx 1 root root 17 Nov  6 01:21 S99diamond -> ../init.d/diamond
lrwxrwxrwx 1 root root 24 Nov  6 01:20 S99libvirt-guests -> ../init.d/libvirt-guests
lrwxrwxrwx 1 root root 11 Sep 16 09:40 S99local -> ../rc.local

Comment 3

[Rail Aliiev [:rail]]

Attachment: swap.diff

Let's move the init script first.

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8518526 [details] [diff] [review]
swap.diff

remote:   https://hg.mozilla.org/build/puppet/rev/e9c0ba28aabf
remote:   https://hg.mozilla.org/build/puppet/rev/52b6de0767c8

Transplanted to production

Comment 5

[Rail Aliiev [:rail]]

The change will show up tomorrow.

Comment 6

[Morgan Phillips [:mrrrgn]]

(In reply to Rail Aliiev [:rail] from comment #5)
> The change will show up tomorrow.

Mind if I take this one over?

Comment 7

[Morgan Phillips [:mrrrgn]]

(In reply to Rail Aliiev [:rail] from comment #5)
> The change will show up tomorrow.

Just because it seems relevant to my other runner bugs btw. :)

Comment 8

[Rail Aliiev [:rail]]

It's all yours! :) Thanks.

Comment 9

[Morgan Phillips [:mrrrgn]]

Attachment: runner.background.diff

This is working in my local tests. I'm also going to pin a centos and ubuntu host to my env to watch it a bit more. My long term thinking here is to make runner itself have a --daemon option.

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8521588 [details] [diff] [review]
runner.background.diff

Review of attachment 8521588 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/runner/templates/runner.initd.erb
@@ +32,5 @@
>  do_start () {
>      echo "Starting runner"
>      logrotate -f /etc/logrotate.d/runner
>      ### XXX -n 1 for testing
> +    if $(nohup ${CMD} -c ${CONFIG} -n 1 ${TASKDIR} >> /var/log/runner.log &); then

I don't think you need to use $(...) here (which is equivalent to `...`). Also, this call is getting less readable...

Can you use LSB's start_daemon/stop_daemon (see http://refspecs.linuxbase.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/iniscrptfunc.html) which should be available on centos and ubuntu after sourcing /lib/lsb/init-functions.

Comment 11

[Morgan Phillips [:mrrrgn]]

We don't have /lib/lsb/init-functions or /sbin/start-stop-daemon on the centos hosts....

Comment 12

[Morgan Phillips [:mrrrgn]]

Attachment: runner.centos.init.diff

We don't have access to the init-functions on all of the centos hosts; but I added simple daemon start/stop behavior here. Tested against my pinned slave. Fixing the centos hosts here will give us the behavior desired, since add_swap is only relevant to those hosts. I think launching with & isn't best practice for upstart in any case.

Comment 13

[Morgan Phillips [:mrrrgn]]

Verified that add_swap is running:

[[e-mail] log]# grep -r "swap" *
boot.log:Enabling /etc/fstab swaps:  [  OK  ]
boot.log:swap_setup: No swap required, 31476352 KB mem available
messages:Nov 14 04:17:35 ip-10-134-49-212 puppet-agent[1402]: (/File[/var/lib/puppet/lib/puppet/parser/functions/swapcase.rb]/ensure) defined content as '{md5}4902f38f0b9292afec66d40fee4b02ec'
messages:Nov 14 04:19:34 ip-10-134-49-212 yum[2522]: Installed: nagios-plugins-swap-1.4.15-2.el6.x86_64
messages:Nov 14 04:20:29 ip-10-134-49-212 puppet-agent[1402]: (/Stage[main]/Tweaks::Swap_on_instance_storage/File[/etc/init.d/add_swap]/ensure) defined content as '{md5}737692e52829d9d7a82a9ee9b48f9378'
messages:Nov 14 04:20:29 ip-10-134-49-212 swap_setup: Creating a swap file /swap_file
messages:Nov 14 04:20:29 ip-10-134-49-212 swap_setup: Fixing permissions
messages:Nov 14 04:20:29 ip-10-134-49-212 swap_setup: Formatting
messages:Nov 14 04:20:29 ip-10-134-49-212 swap_setup: Enabling swap
messages:Nov 14 04:20:29 ip-10-134-49-212 kernel: Adding 4194296k swap on /swap_file.  Priority:-1 extents:9 across:4546552k SS
messages:Nov 14 04:20:29 ip-10-134-49-212 puppet-agent[1402]: (/Stage[main]/Tweaks::Swap_on_instance_storage/Service[add_swap]/ensure) ensure changed 'stopped' to 'running'
messages:Nov 14 14:38:28 ip-10-134-54-109 swap_setup: No swap required, 31476352 KB mem available
yum.log:Nov 14 04:19:34 Installed: nagios-plugins-swap-1.4.15-2.el6.x86_64