262269 - A tool to auto-install missing perl packages on non-Windows systems

Status: RESOLVED FIXED

(Bugzilla :: Installation & Upgrading, enhancement, P1)

Description

[Vlad Dascalu]

Checksetup.pl should offer to auto-install missing perl packages, both on
Windows and Linux.

Before auto-installing a package, a custom message could be printed. For
example, for DBD, we could say in Linux: "make sure you install mysql-devel
before attempting to install this", or something similar. We could offer to do
"urpmi mysql-devel" on behalf of the user as well.

Comment 1

[Frédéric Buclin]

I second this idea. I was talking about this right now on IRC with jouni (jth) and I think checksetup.pl should offer the possibility to install missing packages itself:

  Module MIME-Types is missing. Do you want to install it now? [Y/n]  Y
  OK, now dowloading MIME-Types from CPAN... Done
  Now installing MIME-Types... Done

If we are running Bugzilla on Windows, it would of course use PPM instead of CPAN.

This would save us a lot of time. Max, is that doable for 3.0? This would be a big plus compared to 2.x.

Comment 2

[Max Kanat-Alexander]

Okay, first I'm going to make this happen on *nix, because I have a programmatic interface to CPAN there. There may also be a programmatic interface to PPM, but I don't know it yet.

Comment 3

[Max Kanat-Alexander]

Attachment: install.pl

Okay, here's my work so far, as a separate script. It works very nicely, even for installing modules directly into your Bugzilla directory when you're not root! :-)

Comment 4

[:glob ✱]

> There may also be a programmatic interface to PPM, but I don't know it yet.

  ppm install Some::Package -force -follow
  ppm upgrade Some::Package -install -force -follow

notes:
  - you can't guarantee that the command is non-interactive :(
  - there's also no way to install into a user directory on windows
  - if more than one package matches the name, it won't work (eg Win32::API)

Comment 5

[Max Kanat-Alexander]

Note that after some discussion with LpSolit, we did decide that it would be better to install these modules into a lib/ directory, and include that in all the "use lib" statements in Bugzilla. That way the Bugzilla directory gets a lot less cluttered, and we don't have to worry about making an insanely complex .htaccess to deal with all the new files.

Comment 6

[Steffen Mueller]

(In reply to comment #2)
> Okay, first I'm going to make this happen on *nix, because I have a
> programmatic interface to CPAN there. There may also be a programmatic
> interface to PPM, but I don't know it yet.

Please bear in mind that not all Perl installations on Windows have PPM. Only
the ActiveState distribution has it. For a list of most known distributions,
check out win32.perl.org. In particular, the "Strawberry Perl" distribution,
which comes with a gcc of its own, can be used to install almost anything from
CPAN directly.

To install PPM packages into that distribution, you can resort to stuff like
PAR::Dist::InstallPPD, but that should be considered alpha-quality software.

Cheers,
Steffen

Comment 7

[Max Kanat-Alexander]

CPAN can sometimes throw unexpected errors, and I don't want people to think that it's checksetup itself throwing those errors.

So I'd rather ship this as a separate tool, called cpan-install.pl. Probably contrib/cpan-install.pl, really. I've got it all done, I'll attach it now.

Comment 8

[Max Kanat-Alexander]

Attachment: install-module.pl

Here's the script. It basically makes the whole business of dealing with CPAN easier. It sets all the CPAN parameters for you, and if you aren't root, it installs the modules to a local lib/ directory instead of their normal root position.

Comment 9

[Max Kanat-Alexander]

Comment on attachment 285047 [details]
install-module.pl

I've decided that install-module.pl would be a better name for the script, in case someday it also supports PPM.

Comment 10

[Max Kanat-Alexander]

Attachment: Make checksetup recommend install-module.pl

This patch makes checksetup recommend install-module.pl over directly using CPAN, which should make things a lot easier. Advanced users still will know about CPAN.

Comment 11

[Max Kanat-Alexander]

Attachment: Full Patch, v1

This is a full patch containing everything--the modifications to Bugzilla and the install-module.pl script.

  This version of install-module has been enhanced to work properly with the most modern versions of CPAN (and it still works with the CPAN shipped in perl 5.8.1, too).

  I've also added a --all switch, which will auto-install all of Bugzilla's pre-reqs, including optional modules, for you. 

  The only problem right now is that Authen::Radius asks questions of the user when you install it. You just have to hit enter, but it's kind of annoying. I've emailed the author of Authen::Radius asking for some way to bypass that.

  I technically don't *need* review on this, but I'd like any input I can get.

Comment 12

[Max Kanat-Alexander]

Attachment: v2

I fixed a tiny bug--I wasn't correctly passing $notest to install_module, which is why Authen::Radius was still asking me questions even with modern versions of CPAN. (Current versions of CPAN, such as those shipped with 5.8.5, still will ask questions. We could avoid this by just shipping a CPAN.pm directly in our lib/ directory, if we want.)

Comment 13

[Steffen Mueller]

Hi Max,

(In reply to comment #11)
>   The only problem right now is that Authen::Radius asks questions of the user
> when you install it. You just have to hit enter, but it's kind of annoying.
> I've emailed the author of Authen::Radius asking for some way to bypass that.
> 
>   I technically don't *need* review on this, but I'd like any input I can get.

I had a look at the code and didn't find any glaring bugs.

However, about automating installs: Have a look at CPAN's DistroPrefs. http://search.cpan.org/dist/CPAN/lib/CPAN.pm#Configuration_for_individual_distributions_(Distroprefs)

They're relatively new, but for those users who have a recent CPAN.pm, you could make use of them.

Best regards,
Steffen

Comment 14

[Jochen Wiedmann]

I may be missing something, so fix me if I am wrong, but I am missing something.

If a Bugzilla server has Internet access then using the raw CPAN is already doing a lot for me. In particular, it helps resolving dependencies and doing things like that. In other words, in these cases the install.pl script is helpful, but nothing more.

The worst case scenario are servers without Internet access. I am maintaining no less than three Bugzilla installations for which that is the case. In this scenario by far the most time is spent for resolving dependencies, downloading packages and then doing a manual install (without CPAN). In other words, the ultimate time saver would be, if the install.pl script could be instructed to download all the required modules (plus dependencies) to a directory and later on use a copy of that directory as a cache that allows to run --all without Internet access.

Comment 15

[Frédéric Buclin]

Comment on attachment 285058 [details] [diff] [review]
v2

>Index: install-module.pl

>+    # If we're not root, we have to install the modules into the
>+    # Bugzilla directory.
>+    if (!i_am_root()) {

It would be great if you could also install missing Perl modules into bugzilla/lib/ when you are root, to not interfere with modules installed using distro-specific RPM. It's easier to delete bugzilla/lib/ if something goes wrong than trying to go through /usr/lib/perl5/ to find where these modules have been installed.

Comment 16

[Max Kanat-Alexander]

(In reply to comment #14)
> In other words, the
> ultimate time saver would be, if the install.pl script could be instructed to
> download all the required modules (plus dependencies) to a directory and later
> on use a copy of that directory as a cache that allows to run --all without
> Internet access.

  You could already do that, if you have an identical machine, since it installs to the bugzilla/lib directory if you aren't root. (Identical machine meaning the same Perl build and architecture.)

  You could also do it by placing the tar'ed CPAN modules into your ~/.cpan/ directory.

  But it's definitely a good thought for the future--I know several users who don't have access to the Internet and who would like their lives to be easier.

Comment 17

[Steffen Mueller]

Hi again,

I don't want to be making useless suggestions only, so please tell me when I'm not being helpful.

(In reply to comment #14)
> The worst case scenario are servers without Internet access. I am maintaining
> no less than three Bugzilla installations for which that is the case. In this
> scenario by far the most time is spent for resolving dependencies, downloading
> packages and then doing a manual install (without CPAN). In other words, the
> ultimate time saver would be, if the install.pl script could be instructed to
> download all the required modules (plus dependencies) to a directory and later
> on use a copy of that directory as a cache that allows to run --all without
> Internet access.

I have had this issue before. Not with bugzilla, but with other perl applications. Adam Kennedy has worked on this general tool-chain issue and his approach has yielded pip: http://search.cpan.org/dist/pip/

pip doesn't do the searching and gathering of distribution tarballs, but once cpan-install has determined what to fetch, pip can do the installation part. Perhaps it would need some work to make it install into a local directory. I haven't checked. Adam has always been very responsive to support requests in the past, though.

Comment 18

[Max Kanat-Alexander]

  Hey Steffen. :-) No, your suggestions are very interesting and useful, even if/when I don't use them. You know much, much more about CPAN and packaging than any of us, so whenever you have something to say, please do say it. :-)

Comment 19

[Max Kanat-Alexander]

Attachment: v3

Okay, here's a third version. This version fixes a tiny bug with --all and Template-Toolkit, adds a --local switch, and also installs the DB modules if they're not there. I think there might be a few other changes/bugfixes, too.

Comment 20

[Max Kanat-Alexander]

Attachment: v4

This version allows you to specify multiple module names on the command line.

I've discovered that with a raw perl 5.8.1 installation, install-module.pl --all takes 21 minutes. This makes me think that we should ship CPAN.pm in the lib/ directory in tarballs, because the newer CPANs can do things much faster. (I'm timing how long things take with a new CPAN.pm, right now.)

Comment 21

[Jochen Wiedmann]

(In reply to comment #20)

> --all takes 21 minutes. This makes me think that we should ship CPAN.pm in the
> lib/ directory in tarballs, because the newer CPANs can do things much faster.

Shiver! Please don't! This can only cause trouble. The sensible thing would be to check the version of CPAN.pm and if its lower than "whatever", issue a warning. But we should *never* ship other modules, in particular not CPAN.pm.

Comment 22

[Frédéric Buclin]

I agree, keep other modules out of Bugzilla.

Comment 23

[Max Kanat-Alexander]

(In reply to comment #21)
> Shiver! Please don't! This can only cause trouble. The sensible thing would be
> to check the version of CPAN.pm and if its lower than "whatever", issue a
> warning. But we should *never* ship other modules, in particular not CPAN.pm.

  Why not?

  My tests show that with a CPAN that can do "notest", --all takes only 3 minutes.

  CPAN.pm is a pure-perl module, it's not that hard to ship. We wouldn't be checking it into CVS. I'd just add a line to my tarball-building script that does "install-module.pl CPAN" right before I roll the tarball. This would also keep us protected against future versions of CPAN that break the API that install-module depends on. (CPAN doesn't really *have* a programmer API, so what we're using is technically unstable.)

Comment 24

[John Beranek]

No support for HTTP/FTP proxy settings?

The requirement for this in corporate environments is very common, I believe.

Comment 25

[Mark Bickford]

While I like the overall idea, I'd like to second the call from comment #14 for a way to resolve dependencies and handle boxes without internet access.  I recently upgraded our departmental installation from bz 2.20.x to 3.0.2 (I had still been running MySQL 3.23) without internet access on the bz box, and it took a good part of the day to discern, download, ftp to the server, and install 27 different Perl modules.  Plus, at least at Perl 5.8.0, I have not been able to convince CPAN on our bz server to work with our authenticating proxy, so direct proxy support in your script would be very helpful.

Comment 26

[David Marshall]

Why isn't Bundle::Bugzilla the best option?

Comment 27

[Max Kanat-Alexander]

(In reply to comment #24)
> No support for HTTP/FTP proxy settings?

  No, it does support them. It uses the CPAN configuration that you already have, for those.

(In reply to comment #25)
> I'd like to second the call from comment #14 for
> a way to resolve dependencies and handle boxes without internet access.

   I agree that it would be useful. That would be a separate bug.

> I have not
> been able to convince CPAN on our bz server to work with our authenticating
> proxy, so direct proxy support in your script would be very helpful.

  My script uses the CPAN module, so there's no such thing as "direct proxy support". If your box doesn't work with CPAN, it won't work with this script.

(In reply to comment #26)
> Why isn't Bundle::Bugzilla the best option?

  Because it's only up to date as of the latest stable version, and sometimes not even up to date with that. I'm also not sure if it contains the optional requirements.

Comment 28

[Jochen Wiedmann]

(In reply to comment #27)
> (In reply to comment #26)
> > Why isn't Bundle::Bugzilla the best option?
> 
>   Because it's only up to date as of the latest stable version, and sometimes
> not even up to date with that. I'm also not sure if it contains the optional
> requirements.

Max, if you believe that your script is superior in that aspect: Why not use it to *create* Bundle::Bugzilla? That way we'd always have an uptodate version.

Comment 29

[Max Kanat-Alexander]

(In reply to comment #28)
> Max, if you believe that your script is superior in that aspect: Why not use it
> to *create* Bundle::Bugzilla? That way we'd always have an uptodate version.

  Because Bundle::Bugzilla is only for the latest stable version of Bugzilla, and checksetup.pl and install-module.pl must work all the time, with any development or CVS version, or even an older version of Bugzilla.

Comment 30

[Jochen Wiedmann]

(In reply to comment #29)

>   Because Bundle::Bugzilla is only for the latest stable version of Bugzilla,
> and checksetup.pl and install-module.pl must work all the time, with any
> development or CVS version, or even an older version of Bugzilla.

The *published* Bundle::Bugzila must match the latest stable version. Nothing prevents you from putting a different version into lib.

Comment 31

[Max Kanat-Alexander]

(In reply to comment #30)
> The *published* Bundle::Bugzila must match the latest stable version. Nothing
> prevents you from putting a different version into lib.

  True. But it wouldn't particularly get us anything that the constants in Bugzilla::Install::Requirements aren't already getting us.

Comment 32

[Max Kanat-Alexander]

Attachment: v5

I've separated out the guts of the script into a module, and also moved some useful things into Bugzilla::Install::Util, which allowed a slight cleanup in Bugzilla::Install::Localconfig.

Comment 33

[Max Kanat-Alexander]

Attachment: v6

Okay, I reversed --local to --global. That is, you now have to explicitly specify if you want install-module to install modules globally, even if you're root.

  This also includes POD for bin_loc in Bugzilla::Install::Util.

  As far as I'm aware, this patch is ready for checkin.

Comment 34

[Max Kanat-Alexander]

Comment on attachment 285559 [details] [diff] [review]
v6

Granting myself review as Installation module owner.

Comment 35

[Max Kanat-Alexander]

  I appreciate everybody's feedback. If somebody would like to file a bug about systems that don't have Internet access, and making installation easier for them, that'd be great (unless one already exists).

  I'm going to approve this now and then check it in a bit later when I have the time to do so.

Comment 36

[Max Kanat-Alexander]

RCS file: /cvsroot/mozilla/webtools/bugzilla/install-module.pl,v
done
Checking in install-module.pl;
/cvsroot/mozilla/webtools/bugzilla/install-module.pl,v  <--  install-module.pl
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/CPAN.pm,v
done
Checking in Bugzilla/Install/CPAN.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/CPAN.pm,v  <--  CPAN.pm
initial revision: 1.1
done
Checking in Bugzilla/Install/Filesystem.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Filesystem.pm,v  <--  Filesystem.pm
new revision: 1.24; previous revision: 1.23
done
Checking in Bugzilla/Install/Localconfig.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Localconfig.pm,v  <--  Localconfig.pm
new revision: 1.10; previous revision: 1.9
done
Checking in Bugzilla/Install/Requirements.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Requirements.pm,v  <--  Requirements.pm
new revision: 1.39; previous revision: 1.38
done
Checking in Bugzilla/Install/Util.pm;
/cvsroot/mozilla/webtools/bugzilla/Bugzilla/Install/Util.pm,v  <--  Util.pm
new revision: 1.10; previous revision: 1.9
done
Checking in template/en/default/setup/strings.txt.pl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/setup/strings.txt.pl,v  <--  strings.txt.pl
new revision: 1.4; previous revision: 1.3
done