Closed Bug 316665 Opened 19 years ago Closed 9 years ago

Make Bugzilla work with FastCGI

Categories

(Bugzilla :: Bugzilla-General, enhancement)

2.21
enhancement
Not set
normal

Tracking

()

RESOLVED FIXED
Bugzilla 6.0

People

(Reporter: karl, Unassigned)

References

()

Details

(Keywords: perf, Whiteboard: [roadmap])

Attachments

(1 file, 5 obsolete files)

"FastCGI  is a language independent, scalable, open extension to CGI that provides high performance without the limitations of server specific APIs." -- from the fastcgi.com web site.
In other words, this is similar to mod_perl. There is, however, a difference between mod_perl and FastCGI. mod_perl, from what I understand, caches the compiled script and executes as needed. With fastCGI, the script is executed and run. A big while loop surrounds all of the code that executes once per request. The condition of the loop includes a call which blocks until (a) a request comes in or (b) the server decides to stop the program to conserve resources. There is more, of course. Check the URL for info. From what I can tell, the requirements for mod_perl would also serve well for FastCGI, so it isn't worth doing anything with this until we get mod_perl support.
My understanding is that the requirements for mod_perl and for FastCGI are basically the same, but mod_perl offers better performance improvements and is more of the standard way to go.
There are differneces, but I don't think there are any that would make it easier for us. I have index.cgi and show_bug.cgi working under mp2 locally. (Its rather ugly, but....)
Status: NEW → RESOLVED
Closed: 19 years ago
Resolution: --- → WONTFIX
No longer depends on: mod_perl
I made a slight patch on Bugzilla and it works with my FCGI::Spawn, both 3.2 locally and 3.4 at http://bugs.vereshagin.org Patch is: http://gist.github.com/278799 callout function is a callout parameter for the FCGI::Spawn object, it is executed after every .cgi file included with do().
I forgot to mention that the CGI.pm patch is required for this from the FCGI::Spawn: http://github.com/petr999/fcgi-spawn/blob/master/FCGI-Spawn/CGI.pm.fcgi.spawn.patch
We may actually do this, since FastCGI works with more webservers than mod_perl does.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
Great! Although I'm planning to switch from CGI::Fast to FCGI.pm for accepting connections and therefore the CGI.pm patch shouldn't be necessary. Needless to say how much people themselves use to be unhappy with CGI.pm in favor of CGI::Simple, etc. I'd like to share some comments to the patch: - cleanup of request cache shouldn't be made because of mod_perl, but because of localconfig variable. Or at least in the case any of them are present. Same goes here about the disconnection from database. - environment clearing. This clears some important variables like TZ which is crucial foor Date::Manip. Yes, there should be variable in the localconfig, too. Or, the things that require this feature should be re-done not require this? I'd like to know more on that because the save_env feature of my FCGI::Spawn is just a workaround and thus it's a performance loss. That is all but seems not the everything about why I named Bugzilla a 'worst code ever seen' on the FCGI::Spawn POD, but by now it's all lost in its Git history. It's a no a sweet memory, anyway. ;-) I'm wondering if I'd propose a patch for all of that? although I should only do after I make that CGI::Fast to FCGI.pm transition for FCGI::Spawn anyway.
One possibility for this (a recently-popular modern solution) is: http://plackperl.org/
Yes, Plack/PSGI is a very fashiuonable stuff. Even trendiest Moscow.pm , the source of the codename for Rakudo-1.0 release, the perl6 VM implementation, even they are stil not much aware about it till this moment. For Bugzilla with Plack, there should be any successful use cases and some documantation about the type of the connector, etc. I'm sure that at least Plack's FCGI connector isn't much aware about Bugzilla's specific for existence as a permanent process(es). As well as a 'single actor's theatre', Plack seem to have no care about co-existence of Bugzilla and other applications in the same processes. At the moment being, FCGI::Spawn-0.16.1 need no patching of Buhzilla for its use. But yes, some very special care and even the what is considered to be a dirty hack are made for this, which is a lack for transparency but seem to be a some code hint for next particular application(s) adaptation for bith FCGI::Spawn and any persistent-process handler for Bugzilla like the PSGI/Plack.
WIth FastCGI and MySQL, the Template.pm can't print utf8. THis is because of stricter FCGI./pm newer versions, ~v0.69 They disallow unencoded utf8 ( octets to be printed ). AT the same time, text data representation in all SELECTs apparentlky depends on DBD::Mysql-specific options inside the DBI->connect(). The utf8 enablement should be contained on a DSN line and not among attributes. So here is the patch.
Blocks: bz-rpc-any
Status: REOPENED → NEW
Target Milestone: --- → Bugzilla 4.2
No longer blocks: bz-rpc-any
Attached patch Work In Progress (obsolete) — Splinter Review
Assignee: general → mkanat
Attachment #506369 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attached patch WIP 2 (obsolete) — Splinter Review
This version now runs Bugzilla successfully when you just do: ./app.psgi That actually starts up an HTTP server and everything. It doesn't yet work properly under Apache. It also will probably die if you don't have every single optional module installed. I also need to consolidate duplicate code between app.psgi and mod_perl.pl into a single module, probably something like Bugzilla::Persistent. Or I'll just add a method to Bugzilla.pm.
Attachment #511897 - Attachment is obsolete: true
Attached patch WIP 3 (obsolete) — Splinter Review
I've done some refactoring and made almost everything work under ./app.psgi. xmlrpc.cgi and jsonrpc.cgi don't seem to be working, though.
Attachment #513429 - Attachment is obsolete: true
Attached patch v1 (obsolete) — Splinter Review
Here is a functioning patch, with some caveats. Here's what to know about this: * XML-RPC doesn't work. I don't think there's a way to make SOAP::Lite work with FastCGI. RPC::Any will fix this, and there's already a patch waiting for that. * This does work under Apache, but I would not recommend to Bugzilla users that you run FastCGI Bugzilla instead of mod_perl Bugzilla. There is no way to control the memory usage of FastCGI Bugzilla, and it doesn't share memory between processes under Apache's mod_fcgid, so it uses WAY more memory than mod_perl Bugzilla does, and it can theoretically grow without bound, as there is no standardized way to kill off a process that is taking up too much RAM. Also, the Apache configuration is much more complex than any of our other configuration options. If you want to run it under Apache, here's the config (for mod_fcgid): AliasMatch /fcgi/(.*cgi)$ /var/www/html/fcgi/fastcgi.pl/$1 <Directory /var/www/html/fcgi> <Files 'fastcgi.pl'> SetHandler fcgid-script </Files> Options +Indexes +ExecCGI DirectoryIndex index.cgi AllowOverride Limit FileInfo Indexes </Directory> Where "/fcgi/" is the prefix on your path for URLs and /var/www/html/fcgi/ is the path on the disk where Bugzilla is installed. * I had to add fastcgi.pl to an "allow" in the .htaccess because otherwise the above Apache configuration fails with a "client denied by server configuration". * You can run a standalone Bugzilla without any other webserver, now, by doing "./app.psgi". * FastCGI under Apache is considerably faster than mod_perl, about 3x faster on a simple test (run "ab" against index.cgi with 5 concurrent requests). Quite a bit of this patch (probably most of its bulk) is dedicated to refactoring stuff so that mod_perl and FastCGI can live in harmony. If you want this refactoring separated out into a different patch to make this easier to review, let me know.
Attachment #513681 - Attachment is obsolete: true
Attachment #513717 - Flags: review?(glob)
a plack ppm isn't available for activeperl :( http://code.activestate.com/ppm/Plack/ /me returns to battling strawberryperl
(In reply to comment #16) > a plack ppm isn't available for activeperl :( > http://code.activestate.com/ppm/Plack/ I just reported a bug against Test-TCP so that Plack can hopefully install via PPM: https://rt.cpan.org/Ticket/Display.html?id=66016
Attached patch v2Splinter Review
glob discovered that Extensions were broken under FastCGI (in fact, app.psgi wouldn't even start) because of a chicken-and-egg problem where request_cache wasn't available at the time that load_extensions() was called.
Attachment #513717 - Attachment is obsolete: true
Attachment #514735 - Flags: review?(glob)
Attachment #513717 - Flags: review?(glob)
Whiteboard: [roadmap]
this looks good so far, however i'm going to hold off on an actual review until i can get it working with IIS. ::: app.psgi @@ +61,5 @@ + my $cgi_path = Bugzilla::Constants::bz_locations->{'cgi_path'}; + + my $static_paths = join('|', ALLOWED_STATIC); + enable 'Static', + path => qr{.*/($static_paths)/}, this regex should anchor the paths to the start of the string, not a substring across the whole path. ::: fastcgi.pl @@ +9,5 @@ +my @args = @ARGV; +if ($ENV{PLACK_ENV} and $ENV{PLACK_ENV} eq 'development') { + # XXX Plack daemonizes when you add this argument, which + # doesn't work under mod_fcgid + # push(@args, '-R', "$cgi_dir,$cgi_dir/Bugzilla,$lib_dir"); i don't understand why this code is here :)
(In reply to comment #19) > this regex should anchor the paths to the start of the string, not a substring > across the whole path. Okay. I have that fixed in my local copy. > +if ($ENV{PLACK_ENV} and $ENV{PLACK_ENV} eq 'development') { > + # XXX Plack daemonizes when you add this argument, which > + # doesn't work under mod_fcgid > + # push(@args, '-R', "$cgi_dir,$cgi_dir/Bugzilla,$lib_dir"); > > i don't understand why this code is here :) Ideally, it would be nice, when we're running in development mode, to have Plack reload files automatically if they've changed. However, we can't currently do that under fastcgi because of the way that Plack's reloading system works.
Blocks: 645282
I found out I need to patch the showdependencygraph.cgi. This is because with fastcgi typically there are two users and fastcgi's user may have a very srtict umask. Thus the files created by fastcgi user should be accessible by web server's user if they are to be published. This time it is about the 'webdot' feature: https://gist.github.com/888145 I believe there are many of such a features yet to be adapted for 'fastcgi' feature.
(In reply to comment #15) > * This does work under Apache, but I would not recommend to Bugzilla users > that you run FastCGI Bugzilla instead of mod_perl Bugzilla. There is no way > to control the memory usage of FastCGI Bugzilla, and it doesn't share memory > between processes under Apache's mod_fcgid, so it uses WAY more memory than > mod_perl Bugzilla does, and it can theoretically grow without bound, as there > is no standardized way to kill off a process that is taking up too much RAM. > Also, the Apache configuration is much more complex than any of our other > configuration options. Doesn't this more or less negate the entire reason for even having this patch? I thought the idea was to have a less-invasive alternative to mod_perl to get some performance out of it, but this sounds like it's actually worse instead. Might as well just make people use mod_perl like we have been already.
Also, why using Plack instead of FCGI and/or CGI::Fast? Plack is not available as RPM, at least in Mandriva and RHEL, and it has tons of dependencies (again).
(In reply to comment #22) > Doesn't this more or less negate the entire reason for even having this patch? > I thought the idea was to have a less-invasive alternative to mod_perl to get > some performance out of it, but this sounds like it's actually worse instead. No, the idea is to get people who can't use Apache to also have a fast alternative configuration for Bugzilla. The primary targets are IIS and lighttpd. There will never be a less-invasive solution that mod_perl for Apache--even mod_cgi is more "invasive" in terms of configuration. :-) mod_perl is the ultimate in configuration simplicity for Perl in Apache and will probably always remain that way.
(In reply to comment #23) > Also, why using Plack instead of FCGI and/or CGI::Fast? Plack is not available > as RPM, at least in Mandriva and RHEL, and it has tons of dependencies (again). Yeah, I've thought about this as well. Plack is what the Perl world is standardizing on. There's a *lot* of community support for it, and it's developed very actively. It's not quite perfect yet, and it does have an annoyingly-long dependency tree. It might end up being simpler to just write a simple FastCGI wrapper around Bugzilla, but we'd still have to pull in quite a bit of Plack's infrastructure to wrap our CGIs (all the WrapCGI stuff). So I figured we'd just go with Plack because it's now the "standard." Also, it does a lot more than we are using in this patch and could be really advantageous to us in the future if we want to use some of those extra great features. As far as RPMs go, the present state of a module being packaged or not packaged shouldn't affect our decision to write *future* code for it. That would limit us to modules with pre-existing packages, which is a small fraction of CPAN. (This is true in almost every language, I think, also. Packaged rubygems are a tiny fraction of all available ones, and packaged Python cheeseshop modules also are not the whole cheeseshop.) Modules can always be packaged, and until recently rpmforge was very responsive on this. I expect they will be again when whatever is currently happening gets sorted out. There are a lot of other server platforms too--Windows, Ubuntu, etc. We should care about packaging on all of the popular server platforms, and we should actively assure that each of them are packaging the necessary dependencies to make our users' lives easier.
(In reply to comment #25) > It might end up being simpler to just write a > simple FastCGI wrapper around Bugzilla Well, I would investigate this path first before requiring something as invasive as Plack.
It's certainly a possibility. However, having done the research I'm much more inclined to use a complete library as a good software design practice, rather than write something ourselves. Plack has a lot of dependencies, but it's actually almost completely *un*-invasive, in terms of our need to modify Bugzilla code to work with it. (The modifications I had to do were mostly around having multiple persistent-Perl frameworks.)
Target Milestone: Bugzilla 4.2 → Bugzilla 5.0
It's been a few months since the last post here, and I was wondering what the status is of FastCGI support within Bugzilla? We are moving most of our webservers away from Apache and on to Nginx here, and it's come down to either making Bugzilla work with Nginx, or moving to Trac (which has native FastCGI support).
(In reply to comment #28) > It's been a few months since the last post here, and I was wondering what > the status is of FastCGI support within Bugzilla? I would say that at this point it's stalled. There are some other plans that we will be announcing in the next few months that may eventually fix this, though.
hi bryce If you use Nginx you may find it useful to refer to refer to Bug #645282 . It is not clear if you use FCGI::Spawn which is the only way till this moment to run Bugzilla in FastCGI but if you do you can report troubles on Bugzilla usage with FastCGI to http://bugs.vereshagin.org/product/FCGI%3A%3ASpawn (registration required)(the only Bugzilla known to run in FastCGI/Nginx yet) Be sure to use the latest development snapshot from http://gitweb.vereshagin.org/FCGI-Spawn/shortlog/refs/heads/devel To run the test suite bundled in the package is not a bad idea, too. Though it depends on more Perl modules. Be aware that the comments here may not reflect the current progress of the 'Bugzilla on FastCGI' task as well as a this bug's milestone. Also, you may want to know that FastCGI isn't the 'brand new latest and greatest' of the application protocols available to this date. It just may seem the shortest path to run Bugzilla without Apache (Bug #645282) while having the performance better than CGI (comparable with Apache's mod_perl)
(In reply to Max Kanat-Alexander from comment #29) > (In reply to comment #28) > > It's been a few months since the last post here, and I was wondering what > > the status is of FastCGI support within Bugzilla? > > I would say that at this point it's stalled. There are some other plans > that we will be announcing in the next few months that may eventually fix > this, though. Any updates on this? I used a version of the patch in comment 18 adapted to 4.0.2 to get Bugzilla to run under PSGI on Stackato (a Cloud Foundry based PaaS product from ActiveState): http://www.activestate.com/blog/2011/12/running-bugzilla-cloud So I don't care that much about FastCGI, but would really like to see PSGI support from the main codebase, and not just as an experimental patch attached to a bug report. :)
It would be better to target PSGI than FCGI. If you make ${app} into a PSGI application, then as well as getting FCGI for free, you also get other server connectivity options, for free, plus PSGI middleware options. It may be instructive to note that Catalyst used to have a bunch of server connectivity modules, but moved over to PSGI, so now Catalyst just supports PSGI, and all server connectivity options are handled by Plack adapters. From https://metacpan.org/module/Catalyst::PSGI#SYNOPSIS : "Catalyst used to contain an entire set of Catalyst::Engine::XXXX classes to handle various web servers and environments (e.g. CGI, FastCGI, mod_perl) etc. This has been changed in Catalyst 5.9 so that all of that work is done by Catalyst implementing the PSGI specification, using Plack's adaptors to implement that functionality. This means that we can share common code, and share fixes for specific web servers." Now, you could be all "Hmmm. I don't believe it. I can't be bothered understanding new things. I'm happy with FCGI. Can't we just have FCGI?" If that's the case you might want to ask yourself why all the really clever people keep saying "PSGI's a good idea". Either all the clever people have simultaneously gone nuts, or it really is a good idea.
The status at this point is that you would have to resolve Frédéric Buclin's problems with this patch in some way that he will find acceptable.
(In reply to Max Kanat-Alexander from comment #33) > The status at this point is that you would have to resolve Frédéric > Buclin's problems with this patch in some way that he will find acceptable. It is not clear to me what the problems with the patch are: a) that Plack isn't available as a RHEL rpm? b) that Plack shouldn't be used, but that FCGI or CGI::Fast should be used? c) that Plack prerequisites would be unconditionally added to Bugzilla, even when not used? d) something else? Or is the problem that we are hijacking a FastCGI feature request by asking for PSGI instead? Should this be a separate request?
LpSolit: See above questions.
(In reply to Jan Dubois from comment #34) > It is not clear to me what the problems with the patch are: > > a) that Plack isn't available as a RHEL rpm? A module with so many dependencies should be available as RPM, as install-module.pl won't install it. > b) that Plack shouldn't be used, but that FCGI or CGI::Fast should be used? I said it would be nice to check how hard it would be to use CGI::Fast and/or FCGI as we already require CGI before using Plack. Also, glob's comment 19 is unclear about whether this patch works with IIS or not. If it doesn't, then it must be fixed as mkanat stated in comment 24 that IIS was a targetted web server for FastCGI. > c) that Plack prerequisites would be unconditionally added to Bugzilla, > even when not used? They wouldn't be unconditionally added to Bugzilla as it's an optional module. So I didn't have this in mind. > Or is the problem that we are hijacking a FastCGI feature request by asking > for PSGI instead? Should this be a separate request? No need for a separate bug. This is the same request.
(In reply to Frédéric Buclin from comment #36) > (In reply to Jan Dubois from comment #34) > > It is not clear to me what the problems with the patch are: > > > > a) that Plack isn't available as a RHEL rpm? > > A module with so many dependencies should be available as RPM, as > install-module.pl won't install it. I agree that it would be nice if more Perl modules were available via RPM, but I had no problem installing Plack via install-module.pl (on Linux). All I did was adding this section to Bugzilla::Install::Requirements: { package => 'Plack', module => 'Plack', version => 0.9986, }, Given the huge number of other modules it optionally installs, I don't see why adding Plack is an issue. > > b) that Plack shouldn't be used, but that FCGI or CGI::Fast should be used? > > I said it would be nice to check how hard it would be to use CGI::Fast > and/or FCGI as we already require CGI before using Plack. Also, glob's > comment 19 is unclear about whether this patch works with IIS or not. If it > doesn't, then it must be fixed as mkanat stated in comment 24 that IIS was a > targetted web server for FastCGI. This may all be interesting to the original bug poster. For me it is useless because I'm not interested in FCGI at all, I just want to be able to use PSGI to host Bugzilla (e.g. with nginx and uWSGI on Stackato, or just locally with plackup or starman virtually anywhere). So I would even be fine with dropping the FCGI related bits from the patch and just including the PSGI support in an initial step. So what needs to happen to get PSGI support accepted into core Bugzilla?
Attachment #514735 - Flags: review?(glob)
I checkouted bugzilla trunk source via bzr, but I can't find FastCGI & Plack related code. Where can I find recent patch or code? Or, if I have to use Max Kanat-Alexander's attached v2 patch, then what revision I have to use to apply this patch? Bugzilla with mod-perl is very fast so I don't think PSGI version will be faster than mod-perl one, but sometimes, PSGI version will be very useful for personal use. My office uses mod-perl one but if PSGI will combined into main source tree, I'll recommend PSGI one for people. Also, PSGI version is easy to run without root privilege, and it will be a very cool feature for web hosting user.
This seems to have stalled for no apparent (good) reason. I'd certainly like this change so I can run Bugzilla on nginx. The outstanding issues seems to be: - Plack doesn't have an RPM. Is this really a reason to block the patch? A good many of us don't use RPM-based operating systems to begin with and hopefully anyone who does use one and administers Bugzilla is capable of rolling a RPM or installing Plack from CPAN. - A question on whether Plack supports FastCGI so it may be used on IIS. The answer here is yes - Plack supports FastCGI: http://search.cpan.org/~miyagawa/Plack-0.9989/lib/Plack/Handler/FCGI.pm Could someone tidy and commit the patch now? :)
(In reply to Damien Miller from comment #39) > - Plack doesn't have an RPM. Is this really a reason to block the patch? A > good many of us don't use RPM-based operating systems to begin with and > hopefully anyone who does use one and administers Bugzilla is capable of > rolling a RPM or installing Plack from CPAN. It looks like Fedora added a perl-Plack package in version 16. If a lack of an RPM was a reason I'm not sure it's valid any more. > Could someone tidy and commit the patch now? :) Pretty please?
We are going to branch for Bugzilla 4.4 next week and this bug is either too invasive to be accepted for 4.4 at this point or shows no recent activity. The target milestone is reset and will be set again *only* when a patch is attached and approved. I ask the assignee to reassign the bug to the default assignee if you don't plan to work on this bug in the near future, to make it clearer which bugs should be fixed by someone else.
Target Milestone: Bugzilla 4.4 → ---
Assignee: mkanat → general
Status: ASSIGNED → NEW
Sorry for flaming. I know this should be something simple for someone to do, I can't believe you guys don't have psgi support yet. :/
Eabi: I know this should be something simple to do, so I can't believe you guy has not provided a patch yet here that the maintainers could include. :/
Comment on attachment 514735 [details] [diff] [review] v2 >=== modified file 'Bugzilla/Install/Requirements.pm' >+ { >+ package => 'Plack', >+ module => 'Plack', >+ version => 0, >+ feature => ['fastcgi'], >+ }, From what I could read here and there, Catalyst is the way to go for large installations. Catalyst depends on Plack and Moose, though. Plack doesn't depend on Moose. I'm really not an expert with Plack, Catalyst or any PSGI-related stuff. Has someone with some expertise an opinion on the way to go? >=== modified file 'Bugzilla/Install/Util.pm' >+# These are functions that are used everywhere in Bugzilla, but are >+# needed during times when we can't load Bugzilla::Util. Bugzilla::Util >+# imports them from here and then re-exports them, so you can get them >+# from here or from there. I think it's no longer true that Bugzilla::Util cannot be loaded early during Bugzilla installation/upgrade process, making these moves useless.
In the wake of shellshock, might be a nice idea supporting psgi...
All Bugzilla CGIs run via Perl and not via the shell so shellshock isn't an issue here fortunately. That being said, the lack of progress on this is really frustrating. Bugzilla is the last thing forcing me to keep an Apache instance around.
(In reply to Damien Miller from comment #46) > All Bugzilla CGIs run via Perl and not via the shell so shellshock isn't an > issue here fortunately. Maybe. As long as /bin/sh is not a symlink to bash, or neither Bugzilla nor any of the CPAN modules involved make any system() calls that include shell meta characters.
Getting Bugzilla working *well* under plack is on my personal todo list. We can wrap the cgis in one .psgi as a first step (this has been done repeatedly, and I believe Gerv has even used this for other bugzilla installs) but it's only a first step. I think a reasonable plan of attack would be: 1) PSGI as wrapper, an app.psgi that routes to the (wrapped) CGI scripts. 2) Depend on Plack as a core dependency 3) Any new code that requires an endpoint is a .psgi. The .psgi can run in CGI mode, so nothing is lost. 4) Migrate the CGIs to native PSGI apps, so that app.psgi is just a composition of applications. Rewiring the code so that Bugzilla->cgi is eventually replaced by Plack::Request. Note that these are *my* views, and not representative of other Bugzilla developers or my employer. :)
You can use fcgiwrap to get it to work with Nginx. That's what I've been doing. https://nginx.localdomain.pl/wiki/FcgiWrap
Depends on: 1201113
FYI, I split this bug into two parts: bug 1201113 is about supporting PSGI. It has a patch much smaller than the one proposed here. Once it's fixed, this bug can focus on FastCGI itself.
No longer blocks: 645282
for future installers: supporting PSGI means that you can simply use plackup -s FCGI to spawn a fastcgi server. see bug 645282 for more information on getting it to work with nginx.
Status: NEW → RESOLVED
Closed: 19 years ago9 years ago
Resolution: --- → FIXED
Target Milestone: --- → Bugzilla 6.0
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: