Closed Bug 329556 Opened 18 years ago Closed 17 years ago

Internal Server Error on 60% of all clicks

Categories

(Bugzilla :: Bugzilla-General, defect)

Product:
Bugzilla
Component:
Bugzilla-General
Version:
2.20.1
Platform:
x86
OpenBSD
Type:
defect
Priority:
Not set
Severity:
major

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: phessler, Unassigned)

Details

Attachments

(2 files)

shell script that emulates breakage
497 bytes, text/plain
Details
GDB output after segfault
4.87 KB, text/plain
Details 
User-Agent:       Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8) Gecko/20060108 Firefox/1.5
Build Identifier:  Bugzilla  Version 2.20.1  

After an upgrade from bugzilla 2.18 to 2.20.1, the vast majority (60-70%) of clicks result in 'Internal Server Error'.


Bugzilla 2.20.1, on OpenBSD 3.9-current, apache 1.3.29, mysql 5.0.18,
perl 5.8.6 and mod_perl 1.29.

Reproducible: Sometimes

Steps to Reproduce:
1. New Bug
2. Select any component
3. if it doesn't error, try again.

This is one of several ways to reproduce the bug(s).  All changes to bugs will trigger this, as will general browsing.

Actual Results:  
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, bugzilla@domain.dom and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.



Expected Results:  
Regular bugzilla page.

From 'LogLevel debug' 'ErrorLog logs/bugzilla-error_log' and 'ScriptLog logs/bugzilla-cgi_log'

bugzilla-error_log:
Premature end of script headers: /var/www/bugzilla/htdocs/buglist.cgi

bugzilla-cgi_log:
%% [Mon Mar  6 15:18:50 2006] GET /enter_bug.cgi HTTP/1.1
%% 500 /var/www/bugzilla/htdocs/enter_bug.cgi
%request
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Connection: keep-alive
Cookie: Bugzilla_login=1; DEFAULTFORMAT=specific; Bugzilla_logincookie=499; LASTORDER=bugs.priority%2Cbugs.bug_status%2Cbugs.priority%2Cmap_assigned_to.login_name%2Cbugs.bug_id; BUGLIST=2358%3A2505%3A2087%3A1631%3A1964%3A1965%3A1977%3A2266%3A2384%3A1966%3A1657%3A2874%3A1579%3A1600%3A1613%3A1585%3A1590%3A1577%3A2527%3A2528%3A2529%3A2123%3A2158
Host: bugzilla.domain.dom
Keep-Alive: 300
Referer: http://bugzilla.domain.dom/enter_bug.cgi?product=Product%20Name
User-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8) Gecko/20060108 Firefox/1.5
%response
did you run ./checksetup.pl after upgrading?
naturally.  When I first upgraded, it did fix a number of things and I ran it until everything was reported as hunky dory.

$ ./checksetup.pl

Checking perl modules ...
Checking for       AppConfig (v1.52)   ok: found v1.56
Checking for             CGI (v2.93)   ok: found v3.05
Checking for    Data::Dumper (any)     ok: found v2.121_02
Checking for    Date::Format (v2.21)   ok: found v2.22
Checking for             DBI (v1.38)   ok: found v1.45
Checking for      File::Spec (v0.84)   ok: found v3.01
Checking for      File::Temp (any)     ok: found v0.14
Checking for        Template (v2.08)   ok: found v2.14
Checking for      Text::Wrap (v2001.0131) ok: found v2001.09292
Checking for    Mail::Mailer (v1.65)   ok: found v1.67
Checking for        Storable (any)     ok: found v2.13

The following Perl modules are optional:
Checking for              GD (v1.20)   ok: found v1.41
Checking for     Chart::Base (v1.0)    ok: found v2.3
Checking for     XML::Parser (any)     ok: found v2.34
Checking for       GD::Graph (any)     ok: found v1.43
Checking for GD::Text::Align (any)     ok: found v1.18
Checking for     PatchReader (v0.9.4)  ok: found v0.9.5

Checking user setup ...
Removing existing compiled templates ...
Precompiling templates ...
Checking for      DBD::mysql (v2.9003) ok: found v3.0002
Checking for           MySQL (v3.23.41) ok: found v5.0.18

$


I also ran /sanitycheck.cgi from my webbroswer:
-----
 Bugzilla  Version 2.20.1
Bugzilla Sanity Check
OK, now running sanity checks.

Checking references to classifications.id

... from products.classification_id

Checking references to keyworddefs.id

... from keywords.keywordid

Checking references to fielddefs.fieldid

... from bugs_activity.fieldid

... from profiles_activity.fieldid

Checking references to flagtypes.id

... from flags.type_id

Checking references to bugs.bug_id

... from bugs_activity.bug_id

... from bug_group_map.bug_id

... from attachments.bug_id

... from cc.bug_id

... from longdescs.bug_id

... from dependencies.blocked

... from dependencies.dependson

... from flags.bug_id

... from votes.bug_id

... from keywords.bug_id

... from duplicates.dupe_of

... from duplicates.dupe

Checking references to groups.id

... from bug_group_map.group_id

... from category_group_map.group_id

... from group_group_map.grantor_id

... from group_group_map.member_id

... from group_control_map.group_id

... from user_group_map.group_id

Checking references to profiles.userid

... from profiles_activity.userid

... from profiles_activity.who

... from email_setting.user_id

... from profile_setting.user_id

... from bugs.reporter

... from bugs.assigned_to

... from bugs.qa_contact

... from attachments.submitter_id

... from flags.setter_id

... from flags.requestee_id

... from bugs_activity.who

... from cc.who

... from quips.userid

... from votes.who

... from longdescs.who

... from logincookies.userid

... from namedqueries.userid

... from series.creator

... from watch.watcher

... from watch.watched

... from whine_events.owner_userid

... from tokens.userid

... from user_group_map.user_id

... from components.initialowner

... from components.initialqacontact

Checking references to products.id

... from bugs.product_id

... from components.product_id

... from milestones.product_id

... from versions.product_id

... from group_control_map.product_id

... from flaginclusions.product_id

... from flagexclusions.product_id

Checking references to bug_status.value

... from bugs.bug_status

Checking references to resolution.value

... from bugs.resolution

Checking references to bug_severity.value

... from bugs.bug_severity

Checking references to op_sys.value

... from bugs.op_sys

Checking references to priority.value

... from bugs.priority

Checking references to rep_platform.value

... from bugs.rep_platform

Checking references to series.series_id

... from series_data.series_id

Checking references to series_categories.id

... from series.category

Checking references to whine_events.id

... from whine_queries.eventid

... from whine_schedules.eventid

Checking references to attachments.bug_id / attachments.attach_id

... from flags.bug_id / flags.attach_id

... from bugs_activity.bug_id / bugs_activity.attach_id

Checking references to components.product_id / components.id

... from bugs.product_id / bugs.component_id

... from flagexclusions.product_id / flagexclusions.component_id

... from flaginclusions.product_id / flaginclusions.component_id

Checking references to versions.product_id / versions.value

... from bugs.product_id / bugs.version

Checking references to milestones.product_id / milestones.value

... from bugs.product_id / bugs.target_milestone

... from products.id / products.defaultmilestone

Checking profile logins

Checking cached vote counts

Checking keywords table

Checking cached keywords

Checking for bugs with no creation date (which makes them invisible)

Checking resolution/duplicates

Bug(s) found marked resolved duplicate and not on duplicates table: [list of bugs] (as buglist)

Checking statuses/resolutions

Checking statuses/everconfirmed

Bugs with confirmed status but don't have everconfirmed set: [list of bugs] (as buglist)

Checking votes/everconfirmed

Bugs that have enough votes to be confirmed but haven't been: [list of bugs] (as buglist)

Checking dates in groups.last_changed

Checking dates in profiles.refreshed_when

Checking for bad values in group_control_map

Checking for bugs with groups violating their product's group controls

Checking for unsent mail

Bugs that have changes but no mail sent for at least half an hour: [list of bugs] (as buglist)

Send these mails.

Sanity check completed. 
-----
check apache's error log file for the reason of the errors.
Like I said in my report comment:

"Premature end of script headers: /var/www/bugzilla/htdocs/buglist.cgi"

No reason is given.  That is from the file created with 
'ErrorLog logs/bugzilla-error_log'
this is really a support issue, however what happens when you run buglist.cgi from the command line?
$ ./buglist.cgi
Refresh: 10; URL=query.cgi
Content-Type: text/html

[Tue Mar  7 08:22:01 2006] buglist.cgi: Use of uninitialized value in pattern match (m//) at (eval 27) line 4.
[Tue Mar  7 08:22:01 2006] buglist.cgi: Use of uninitialized value in pattern match (m//) at (eval 27) line 4.
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
                      "http://www.w3.org/TR/html4/loose.dtd">
<html>
  <head>
    <title>Parameters Required</title>
...


I'm not sure what you mean by "support issue"?
You mention mod_perl... Bugzilla behaves strangely with it. Maybe that's the reason. What happens if you try and disable mod_perl?
Same results with mod_perl deactivated

logs/bugzilla-error_log:
 "Premature end of script headers: /var/www/bugzilla/htdocs/enter_bug.cgi"

logs/bugzilla-cgi_log:
%% [Tue Mar  7 10:45:00 2006] GET /enter_bug.cgi?product=Product%20Name HTTP/1.1
%% 500 /var/www/bugzilla/htdocs/enter_bug.cgi
%request
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Connection: keep-alive
Cookie: DEFAULTFORMAT=specific; Bugzilla_logincookie=499; LASTORDER=bugs.priority%2Cbugs.bug_status%2Cbugs.priority%2Cmap_assigned_to.login_name%2Cbugs.bug_id; BUGLIST=2358%3A2505%3A2087%3A1631%3A1964%3A1965%3A1977%3A2266%3A2384%3A1966%3A1657%3A2874%3A1579%3A1600%3A1613%3A1585%3A1590%3A1577%3A2527%3A2528%3A2529%3A3620%3A2123%3A2158; Bugzilla_login=1
Host: bugzilla.domain.dom
Keep-Alive: 300
Referer: http://bugzilla.domain.dom/enter_bug.cgi
User-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8) Gecko/20060108 Firefox/1.5
%response


FWIW, 2.18 ran just fine with mod_perl.
I have been able to reproduce it with a really cheesy shell script.

I copied the data from bugzilla-cgi_log, and wrapped it in an 'echo' thusly:

...
#!/bin/sh
echo 'Accept: Text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Connection: keep-alive
Cookie: Bugzilla_logincookie=499; LASTORDER=relevance%20desc; Bugzilla_login=1; DEFAULTFORMAT=specific
Host: bugzilla.sf.coverity.com
Keep-Alive: 300
User-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8) Gecko/20060108 Firefox/1.5

Hello World
'
...

That works as expceted.  But when I modify the script thusly:

...
#!/bin/sh


echo 'Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Connection: keep-alive
Cookie: Bugzilla_logincookie=499; LASTORDER=relevance%20desc; Bugzilla_login=1; DEFAULTFORMAT=specific
Host: bugzilla.sf.coverity.com
Keep-Alive: 300
User-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8) Gecko/20060108 Firefox/1.5'
...

(notice the difference at the end), it errors out exactly the same way as the scripts.  I think the scripts are getting confused and aren't printing anything other than the httpd headers to STDOUT or STDERR.

Any ideas for tracking this down more?
This is a simple shell script that emulates the breakage.  put it in your bugzilla directory, and browse to it.  *boom*
(In reply to comment #8)
> FWIW, 2.18 ran just fine with mod_perl.

No, it didn't.  Trust me. :)  mod_perl has to be enabled per-directory.  It's probably likely that you never had it enabled for the directory Bugzilla resides in anyway.

Open globals.pl in an editor and look for the following: (it's a couple pages down from the top of the file)

# The following subroutine is for debugging purposes only.
# Uncommenting this sub and the $::SIG{__DIE__} trap underneath it will
# cause any fatal errors to result in a call stack trace to help track
# down weird errors.
#sub die_with_dignity {
#    use Carp;  # for confess()
#    my ($err_msg) = @_;
#    print $err_msg;
#    confess($err_msg);
#}
#$::SIG{__DIE__} = \&die_with_dignity;

Do as the comment says, and uncomment the sub and the line under it.  Then run it and see if you get more detail either in the browser or the error log.
Er, by "run it" I mean browse to some of the files that were generating errors before.
Still nothing from bugzilla-error_log, but bugzilla-cgi_log has some new information:

...
%% [Wed Mar  8 11:07:02 2006] GET /buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=&content=openbsd&serverpush=0 HTTP/1.1
%% 500 /var/www/bugzilla/htdocs/buglist.cgi
%request
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip,deflate
Accept-Language: en-us,en;q=0.5
Connection: keep-alive
Cookie: DEFAULTFORMAT=specific; Bugzilla_logincookie=499; Bugzilla_login=1; LASTORDER=relevance%20desc
Host: bugzilla.domain.dom
Keep-Alive: 300
User-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.8) Gecko/20060108 Firefox/1.5
%response
%stderr
Subroutine main::confess redefined at /usr/libdata/perl5/Exporter.pm line 65.
 at globals.pl line 104
Subroutine main::croak redefined at /usr/libdata/perl5/Exporter.pm line 65.
 at globals.pl line 104
Subroutine main::carp redefined at /usr/libdata/perl5/Exporter.pm line 65.
 at globals.pl line 104

...

stdout isn't shown, so I think the script just isn't printing anything at all.  I've tried reproducing it from the command line, but no dice.
Upgraded to 2.20.2, re-ran checksetup.pl and sanitycheck.cgi, still broken in the same way.
Tried out 2.22 in my test env.  Can nolonger reproduce it with the 'new product' test.  however, every click on 'user prefs' will trigger it.  unknown if anything else will trigger it.  test env is identical to original system..
There seems to be a problem with DBD::mysql when running on OpenBSD 3.9.  I think that this is probably the cause of this particular bug.  It's a problem with DBD::mysql, not with OpenBSD.  DBD::mysql causes seemingly random perl segfaults.  I found this out by gdb'ing a perl.core I found in my bugzilla installation directory.  Then I googled and found this:

http://rt.cpan.org/Public/Bug/Display.html?id=18884

OpenBSD ships with version 3.0002.  I upgraded to 3.0006 through CPAN and it seems to have reduced the frequency of the segfaults, but it has not eliminated them completely.  It still happens periodically.

Presumably, this is happening on OpenBSD due to its stack/heap protection mechanisms (propolice, etc.).

One workaround is to use PostgreSQL instead of mysql (and therefore DBD::Pg), although the bugzilla docs say that postgres has not been tested as extensively as mysql.

Steps to reproduce:

Take Bugzilla 2.22 on OpenBSD 3.9 using mysql 5.0.22 and DBD::mysql 3.0002 or 3.0006 (all required perl modules were installed using pkg_add, where possible). 

Run one of the .cgi scripts from the shell repeatedly.

Repeat until segfault.

Examine core with gdb:
# gdb /usr/bin/perl
(gdb) target core perl.core
(gdb) bt
Here's the output of GDB for anyone that wants to look at DBD::mysql but doesn't have OpenBSD to reproduce.
http://rt.cpan.org/Ticket/Display.html?id=20868

Filed in CPAN DBD::mysql bug tracker as well.
The bug does seem to live in DBD::mysql.

*HOWEVER* (in my environment) only Bugzilla triggers it.

http://lists.mysql.com/perl/3933 has a reproducable testcase and workaround.
Marking the bug as WFM as it seems to be a bug in DBD::mysql. I don't see what we could do about it.
Status: UNCONFIRMED → RESOLVED
Closed: 17 years ago
Resolution: --- → WORKSFORME
Version: unspecified → 2.20.1
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: