979441 - Under mod_perl, some modules aren't preloaded at startup

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, defect)

Description

[Philippe M. Chiasson (:gozer)]

Attachment: bugzilla-startup-load-modules.diff

Generally speaking, it's a good idea to preload as many Perl modules as possible during startup, this way, the code and memory used is shared across all children process.

In the case of Bugzilla, it does some of that via mod_perl.pl but looks like some things are being missed.

Following patch alters startup to report modules loaded at run-time.

This means there should probably a review of this in production and the proper modules added to mod_perl.pl

In my test setup, for instance, it reported this after hitting show_bug.cgi once:

[Tue Mar 04 13:16:56.001132 2014] [:warn] [pid 32457] [32457] Package Bugzilla::Auth::Login::CGI loaded at run-time
[Tue Mar 04 13:16:56.001161 2014] [:warn] [pid 32457] [32457] Package Bugzilla::Auth::Verify::DB loaded at run-time
[Tue Mar 04 13:16:56.001172 2014] [:warn] [pid 32457] [32457] Package Bugzilla::DB::Mysql loaded at run-time
[Tue Mar 04 13:16:56.001178 2014] [:warn] [pid 32457] [32457] Package Bugzilla::DB::Schema::Mysql loaded at run-time
[Tue Mar 04 13:16:56.001201 2014] [:warn] [pid 32457] [32457] Package Bugzilla::Template::Context loaded at run-time
[Tue Mar 04 13:16:56.001207 2014] [:warn] [pid 32457] [32457] Package Bugzilla::Template::Plugin::Bugzilla loaded at run-time
[Tue Mar 04 13:16:56.001223 2014] [:warn] [pid 32457] [32457] Package Bugzilla::Template::Plugin::Hook loaded at run-time
[Tue Mar 04 13:16:56.001231 2014] [:warn] [pid 32457] [32457] Package Bugzilla::User::Setting::Lang loaded at run-time
[Tue Mar 04 13:16:56.001237 2014] [:warn] [pid 32457] [32457] Package Bugzilla::User::Setting::Skin loaded at run-time
[Tue Mar 04 13:16:56.001252 2014] [:warn] [pid 32457] [32457] Package Bugzilla::User::Setting::Timezone loaded at run-time
[Tue Mar 04 13:16:56.001264 2014] [:warn] [pid 32457] [32457] Package CGI::Cookie loaded at run-time
[Tue Mar 04 13:16:56.001271 2014] [:warn] [pid 32457] [32457] Package Cache::Memcached loaded at run-time
[Tue Mar 04 13:16:56.001277 2014] [:warn] [pid 32457] [32457] Package Cache::Memcached::GetParser loaded at run-time
[Tue Mar 04 13:16:56.001295 2014] [:warn] [pid 32457] [32457] Package Compress::Zlib loaded at run-time
[Tue Mar 04 13:16:56.001304 2014] [:warn] [pid 32457] [32457] Package DBD::mysql loaded at run-time
[Tue Mar 04 13:16:56.001319 2014] [:warn] [pid 32457] [32457] Package DBI::Const::GetInfo::ANSI loaded at run-time
[Tue Mar 04 13:16:56.001325 2014] [:warn] [pid 32457] [32457] Package DBI::Const::GetInfo::ODBC loaded at run-time
[Tue Mar 04 13:16:56.001344 2014] [:warn] [pid 32457] [32457] Package DBI::Const::GetInfoType loaded at run-time
[Tue Mar 04 13:16:56.001358 2014] [:warn] [pid 32457] [32457] Package DateTime::TimeZone::America::New_York loaded at run-time
[Tue Mar 04 13:16:56.001374 2014] [:warn] [pid 32457] [32457] Package DateTime::TimeZone::Local::Unix loaded at run-time
[Tue Mar 04 13:16:56.001390 2014] [:warn] [pid 32457] [32457] Package DateTime::TimeZone::OlsonDB loaded at run-time
[Tue Mar 04 13:16:56.001397 2014] [:warn] [pid 32457] [32457] Package DateTime::TimeZone::OlsonDB::Change loaded at run-time
[Tue Mar 04 13:16:56.001403 2014] [:warn] [pid 32457] [32457] Package DateTime::TimeZone::OlsonDB::Observance loaded at run-time
[Tue Mar 04 13:16:56.001419 2014] [:warn] [pid 32457] [32457] Package DateTime::TimeZone::OlsonDB::Rule loaded at run-time
[Tue Mar 04 13:16:56.001424 2014] [:warn] [pid 32457] [32457] Package DateTime::TimeZone::OlsonDB::Zone loaded at run-time
[Tue Mar 04 13:16:56.001453 2014] [:warn] [pid 32457] [32457] Package HTML::Scrubber loaded at run-time
[Tue Mar 04 13:16:56.001474 2014] [:warn] [pid 32457] [32457] Package IO::Uncompress::Adapter::Inflate loaded at run-time
[Tue Mar 04 13:16:56.001480 2014] [:warn] [pid 32457] [32457] Package IO::Uncompress::Base loaded at run-time
[Tue Mar 04 13:16:56.001485 2014] [:warn] [pid 32457] [32457] Package IO::Uncompress::Gunzip loaded at run-time
[Tue Mar 04 13:16:56.001491 2014] [:warn] [pid 32457] [32457] Package IO::Uncompress::RawInflate loaded at run-time
[Tue Mar 04 13:16:56.001512 2014] [:warn] [pid 32457] [32457] Package Socket6 loaded at run-time
[Tue Mar 04 13:16:56.001519 2014] [:warn] [pid 32457] [32457] Package String::CRC32 loaded at run-time
[Tue Mar 04 13:16:56.001537 2014] [:warn] [pid 32457] [32457] Package Template::Namespace::Constants loaded at run-time
[Tue Mar 04 13:16:56.001558 2014] [:warn] [pid 32457] [32457] Package Template::Plugin loaded at run-time

Comment 1

[:glob ✱]

a fair number of these are because they are conditionally loaded by bugzilla or dependencies.

instead of trying to pre-determine the list, it should be possible to simply track modules which have been loaded at runtime in a file, and then pre-load those modules the next time the worker restarts.

Comment 2

[:glob ✱]

Attachment: 979441_1.patch

here's the changes to mod_perl which implement the "record what is loaded at run-time, preload that next time we startup" approach.

this should go upstream, but i'd like to measure the difference this makes before pushing it upstream.

if this gets an r+ i intend on committing it to bugzilla-dev and letting it run there for a while.

Comment 3

[Philippe M. Chiasson (:gozer)]

Comment on attachment 8450030 [details] [diff] [review]
979441_1.patch

Looks good to me!

Comment 4

[Dylan Hardison [:dylan] (he/him)]

Comment on attachment 8450030 [details] [diff] [review]
979441_1.patch

Review of attachment 8450030 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good to me, I don't notice a change in performance though.

::: mod_perl.pl
@@ +38,4 @@
>  use Apache2::Log ();
>  use Apache2::ServerUtil;
>  use ModPerl::RegistryLoader ();
> +use File::Slurp ();

I see we use this in other files, but I don't see it in the list of REQUIRED_MODULES or OPTIONAL_MODULES and it isn't in corelist. Why is that? We must implicitly depend on it via some other module as it is always installed...

@@ +65,5 @@
> +    chomp(@files);
> +    foreach my $file (@files) {
> +        $preloaded_files{$file} = 1;
> +        Bugzilla::Util::trick_taint($file);
> +        eval { require $file };

nit: eval { require $string } instead of do $string

Comment 5

[Dylan Hardison [:dylan] (he/him)]

Should I change the status of this bug to ASSIGNED and set approval? and Milestone? I can't do either because it is NEW.

Comment 6

[:glob ✱]

(In reply to Dylan William Hardison [:dylan] from comment #4)
> ::: mod_perl.pl
> @@ +38,4 @@
> >  use Apache2::Log ();
> >  use Apache2::ServerUtil;
> >  use ModPerl::RegistryLoader ();
> > +use File::Slurp ();
> 
> I see we use this in other files, but I don't see it in the list of
> REQUIRED_MODULES or OPTIONAL_MODULES and it isn't in corelist. Why is that?
> We must implicitly depend on it via some other module as it is always
> installed...

good catch; i'll add it in.

(In reply to Dylan William Hardison [:dylan] from comment #5)
> Should I change the status of this bug to ASSIGNED and set approval? and
> Milestone? I can't do either because it is NEW.

no; we don't do approvals or milestones on bmo bugs.

Comment 7

[:glob ✱]

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   ab5ac55..f8e9d66  development -> development

Comment 8

[:glob ✱]

To ssh://[e-mail]/webtools/bmo/bugzilla.git
   3431a53..c7a6d5e  master -> master