95082 - Param('shutdownhtml') doesn't completely shut down Bugzilla

Status: RESOLVED FIXED

(Bugzilla :: Bugzilla-General, defect, P2)

Description

[Jacob Steenhagen]

Entering something into shutdownhtml doesn't completely disable Bugzilla as
there are some queries that are run before bugzilla calls PutHeader() [which is
where shutdownhtml takes effect].  This causes issues with the number of
available connections when bugzilla is trying to do things like resync the
shadow db.  Also, when bugzilla is shut down, you see all the normal page
headers (FE, the last time bugzilla was shut down I was in the middle of making
a change and the page header said that the bug was processed, but immediatly
underneath that it said bugzilla was down).

Comment 1

[Jacob Steenhagen]

As much as I hate to add something else to 2.14, this causes issues when
mozilla.org is attemping to recover from unplanned downtime.  It may also be
contributing to the failure of resyncing the shadow database.

Comment 2

[Matthew Tuck [:CodeMachine]]

Well, if syncshadowdb actually did shut down Bugzilla when they resyncing the
database, that might be right.  But they don't, the backup just locks the
tables.  See bug #75840.

Comment 3

[Jacob Steenhagen]

That's true and it is something to be addressed.  I don't know for sure if this
is what's causing the nightly resync to fail, but it seems likely.  If someone
else agrees that this is the likely cause, we should probably up the target for
bug 75840 to 2.14 as it is causing mozilla.org many headaches.

The bulk of what this report is about was discovered when Dawn manually put some
text in shutdownhtml this morning so she could rebuild the shadow database.  She
had to shut down httpd in order to kill all the stray MySQL connections that
kept appearing (she was getting 'too many users' errors).

Comment 4

[Matthew Tuck [:CodeMachine]]

As far as I can tell the nightly process locks down the database just fine. 
Maybe there is a bug there that you can still access certain tables, but I doubt
it.

If that was the case, the correct fix is to lock the tables down properly, not
to shutdown Bugzilla.  For databases of sufficiently small size, shutting down
is not appropriate and the queries can just wait for a minute or two.

As far this bug, it's certainly a bug, but my assertion was the two have no
relationship and hence this isn't a 2.14 blocker.

Comment 5

[Jacob Steenhagen]

Attachment: patch

Comment 6

[Jacob Steenhagen]

When making this patch, I went under the assumption that when bugzilla is shut
down, the only pages that should be accessed are editparams.cgi and
doeditparams.cgi (it is possible for the administrator to log into bugzilla
using only editparams.cgi).  It looks like that's what $ignoreshutdown was
attempting to do, but it also allowed access to any page that required a login
(as logging in was considered a reason to ignore the shutdown param).  To that
end, this patch will not send anything to the SQL server unless the page being
viewed is editparams.cgi or doeditparams.cgi.  It also always sets the page
title to "Bugzilla is Down" when there is something in the shutdownhtml param.

Comment 7

[Jacob Steenhagen]

r=justdave in IRC.

Checked in.

Comment 8

[Jacob Steenhagen]

Was looking at the code for syncshadowdb (re bug 75840) and relized that this
patch would prevent the shadowdb from being rebuilt if bugzilla is shut down. 
The fix is easy, we just have to tell SendSQL() that it's OK to pass on the
query if the command being run is syncshadowdb.

Comment 9

[Jacob Steenhagen]

Attachment: bugfix

Comment 10

[Matthew Tuck [:CodeMachine]]

I would expect that it wouldn't run if Bugzilla is shut down.

Comment 11

[Dave Miller [:justdave]]

r=justdave
it's in.

Comment 12

[Dave Miller [:justdave]]

Moving to Bugzilla product