Jobqueue.pl forks more than twice and so cannot be properly stopped by Upstart in Ubuntu

UNCONFIRMED
Unassigned

Status

()

Bugzilla
Email Notifications
UNCONFIRMED
4 years ago
3 years ago

People

(Reporter: Mike Budd, Unassigned)

Tracking

Details

(Whiteboard: WONTFIX?)

(Reporter)

Description

4 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0 (Beta/Release)
Build ID: 20140716183446

Steps to reproduce:


1. On Ubuntu 12.04 running Bugzilla 4.4, I followed the instructions in Jobqueue documentation to install Jobqueue as a system daemon, and the instructions on how to do this in the Upstart Cookbook. I created an Upstart jobqueue.conf in the /etc/init directory to start jobqueue.pl using 'start on started JOB=mysql', similarly to 'stop on stopped JOB=mysql, telling Upstart to 'expect daemon' and 'respawn' if process closed.  I also tried using the standard startup and shutdown run levels as triggers: 'start on runlevel [2345]' 'stop on runlevel [016]'. (The more natural trigger Apache cannot be easily used as Apache uses an Init.d script).

2. Having experience the Ubuntu shutdown hang (see below), I decided to go through the 'Upstart cookbook' to see what could be wrong and found the source could be that the number of forks the daemon makes was mis-specified. Unfortunately Upstart cannot handle more than two forks. I then found this Upstart article that talks about an Upstart bug that hangs shutdown if the number of forks is mis-specified: http://stackoverflow.com/questions/12200217/can-upstart-expect-respawn-be-used-on-processes-that-fork-more-than-twice. That references this Upstart bug trace which says it's a deep problem and still unfixed and gives a number of workarounds - but they seem a bit nasty to me: https://bugs.launchpad.net/upstart/+bug/406397

3. I ran the following to find out the number of forks for jobqueue.pl:
root@ubuntu:/var/www/bugzilla# sudo strace -o /tmp/strace.log -fFv /var/www/bugzilla/jobqueue.pl start &
sudo strace -o /tmp/strace.log -fFv /var/www/bugzilla/jobqueue.pl
root@ubuntu:/var/www/bugzilla# sudo killall -9 strace
root@ubuntu:/var/www/bugzilla# sudo egrep "\<(fork|clone)\>\(" /tmp/strace.log | wc | awk '{print $1}'

4. I decided to work round this by starting jobqueue.pl from the upstart script in non-daemon mode using the -f switch. The exact exec line that works is: exec sudo -u root -p password /var/www/bugzilla/jobqueue.pl -f start &



Actual results:

1. In all cases Ubuntu hung for c 15 mins on each shutdown
2. N/A
3. The number of forks was 4!
4. Ubuntu no longer hangs


Expected results:

1. No hang, because Jobqueue.pl should probably only fork twice. Further forks are not normally necessary for a daemon.
(Reporter)

Updated

4 years ago
Summary: Jobqueue.pl forks more than twice and so cannot be properly started and stopped by Upstart in Ubuntu → Jobqueue.pl forks more than twice and so cannot be properly sstopped by Upstart in Ubuntu
(Reporter)

Updated

4 years ago
Summary: Jobqueue.pl forks more than twice and so cannot be properly sstopped by Upstart in Ubuntu → Jobqueue.pl forks more than twice and so cannot be properly stopped by Upstart in Ubuntu
(Reporter)

Updated

4 years ago
OS: All → Linux

Comment 1

4 years ago
Did you take a look at contrib/bugzilla-queue.rhel or contrib/bugzilla-queue.suse? Maybe they will help you fix your problem.
Assignee: extensions → email-notifications
Component: Extensions → Email Notifications
(Reporter)

Comment 2

4 years ago
Thanks Frederik for your suggestions. In fact the -f switch solves the Upstart hanging problem though I realize that is a bit of a hack. Is it likely to cause other issues?

There are other things however that make it difficult to create an Upstart script. Dependency should probably be on Bugzilla being live (inc not shutdown via Shutdownhtml) but there is no Bugzilla daemon .... etc. I have a work around that seems to work, and will create a wish for a proper installation script to be supplied with jobqueue that works on modern Linux installations and offer my files as a initial contribution, if that would be helpful.
the fork count shown by strace is probably misleading -- jobqueue works by having a parent process which monitors the queue for messages to deliver.  when messages are found, a child process is forked to process the queue, which terminates when the queue is empty.

there's no issue with using -f.  this only impacts the startup forking behaviour; worker processes are still forked to manage the queue.

given http://upstart.ubuntu.com/cookbook/#expect has
> If your daemon has a "don't daemonize" or "run in the foreground" mode,
> then it's much simpler to use that and not run with fork following
this bug is most likely a WONTFIX.
Whiteboard: WONTFIX?
(Reporter)

Comment 4

4 years ago
(In reply to Byron Jones ‹:glob› from comment #3)
> the fork count shown by strace is probably misleading -- jobqueue works by
> having a parent process which monitors the queue for messages to deliver. 
> when messages are found, a child process is forked to process the queue,
> which terminates when the queue is empty.
> 

Interesting, though I wonder if that was the case. Please note the server had just been rebooted and Jobqueue was stopped when I ran the above script which itself started the Jobqueue process, and the server was not being used much when I tested (between builds). Updates are maybe 3-4 time a day. So it would be a bit of a co-incidence, though possible, if it immediately forked a child process and that child process would then have to fork another one to give a count of 4 including the initial daemon forks? (Not sure of my Linux here - bit of a newbie, sorry).

Anyway, just wondering, if that is not the problem then why does Jobqueue run by Upstart without the -f and with the 'expect daemon' switch hang Ubuntu 12.04 on shutdown, and with the -f it does not? It is reliably replicable on my machine, and there cannot always be a live child process on shutdown. Can anyone else replicate?

> there's no issue with using -f.  this only impacts the startup forking
> behaviour; worker processes are still forked to manage the queue.
> 
> given http://upstart.ubuntu.com/cookbook/#expect has
> > If your daemon has a "don't daemonize" or "run in the foreground" mode,
> > then it's much simpler to use that and not run with fork following
> this bug is most likely a WONTFIX.

Overall it seems to me that Jobqueue is a pretty critical part of a practical Bugzilla installation, and so needs an installation script that reliably installs it as a system process on modern linuxes like Ubuntu. We do global updates from time to time, and have lots of email subscriptions per bug, so I hate to think how long it would take without Jobqueue! (There is however I think a valid separate question about whether it should be an Upstart, Init.d, or System D script/config or all 3) I'll make a separate wish about that with a sample script if you like.

As you say whether that means this bug should be fixed depends on the implications of running with the -f switch. I though running daemons non-forked was frowned upon for reasons other than Upstart compatibility, and guessed that was for good reasons, but I do not have enough linux to know what those are, so am willing to be advised if the -f switch is entirely innocuous. :)
You need to log in before you can comment on or make changes to this bug.