44659 - [meta] Makefile may help install, quiz program?

Status: RESOLVED WONTFIX

(Bugzilla :: Installation & Upgrading, enhancement, P1)

Description

[Rémi Perrot]

It would be usefull to have a makefile to help at install time.
It may resolve the perl path by simple rule for example.

I'm working on it as the first step to a debian package since it can go in
the main stream while MySQL is now GPL.

Comment 1

[Rémi Perrot]

Attachment: Makefile to install bugzilla

Comment 2

[Chris Yeh]

evaluate for 2.12

Comment 3

[Chris Yeh]

i mis typed the status whiteboard, so this didn't show up in any of my list 
queries.

Comment 4

[Zach Lipton [:zach]]

I think that a quiz program would help the install even more. I have seen 
some programs that do this, where it would do the standard checking 
stuff, and then instead of quitting and making you edit localconfig, you just 
plug in the information and it makes the localconfig file for you. That way, 
you only run the program once, and it will do everything for you. I'm not 
sure what amount of work this requires. Adding a ? after the 2.12

Comment 5

[Andreas Franke (gone)]

Ok, it seems to me that we really have a case of excessive voting here... :)
Anyway, I like the idea of a makefile, and the possibility of a "quiz" program
should not prevent this idea from being realized in 2.12.

How would a "makefile installation" interact with bugzilla upgrades? Wouldn't
this make upgrading more difficult, especially since nearly everyone has
site-specific modifications? Or would it make it even easier?

Another question: would it be possible to create a makefile for windows that
does the same?

Comment 6

[Gervase Markham [:gerv]]

I'd say this is too big a change for a point release. If it ain't broke...

Gerv

Comment 7

[Rémi Perrot]

If it's to big for this point of release why not split it in smallest task.
1 - A static install tool ( Makefile ? )
2 - An install tool builder ( A la configure )
3 - Make the install tool builder support upgrade
(someware between the above task) - Windows stuff, Mac, Amiga, Atari, Beos, Hurd

It's very ugly to have cgi, static page, tools, and data file in the same
directory (it's not FHS compilien) but without installer it's very difficult to
improve this.

Never mind if the first steep is not ready for this release but it must be done.

Comment 8

[Chris Yeh]

Just something to point out: people are trying to install Bugzilla on WinNT and 
make isn't available on that platform unless you install it.

It would be cool if there was a configure.pl or an install.pl script that asked 
you the right questions and did everything.

I don't think this is going to make a 2.12, so I'm going to drop it from the 
list for now. I agree that installation and configuration need to be easier.

Comment 9

[Dave Miller [:justdave]]

Moving to real milestones...

Comment 10

[Hixie (not reading bugmail)]

Taking all Bugzilla 3.0 bugs -- congratulations to MattyT for the triage, it
really was spot on. Note: I may end up pushing some of these bugs back to 3.2,
we'll see. However, I believe all these bugs should just fall out of the 
redesign. Let's hope I'm right!

(Note: I'm also resetting the priority field to "---" so that I can retriage any
that I consider important or likely to be dropped.)

Comment 11

[Dave Miller [:justdave]]

Moving this to 2.16...  debian.org is forcing the issue, and if we don't do it 
soon, they'll do it for us (or may have already done so by now)

And I *really* don't like the way they're doing it, no offense intended, Rémi.

debian.org wants to make bugzilla FHS-compliant (as noted in earlier comments on 
this bug and in recent emails that I've been CCed on).

According to the emails, this is what they want to do:

  - static html and js and gif -> /usr/share/bugzilla/web/
  - cgi                        -> /usr/lib/cgi-bin/
  - modules and .pl            -> /usr/share/bugzilla/lib/
  - localconfig                -> /etc/bugzilla/
  - data/*, graph/*, shadow/*  -> /var/lib/bugzilla/

Now it may just be that some of these issues I'm about to mention just hadn't 
been thought of when they were deciding on this structure...  If you have 
suggestions for how to do this without removing these features I'm open to hear 
them.

My comments on this:

1) Dumping all of Bugzilla's CGI files into the common cgi-bin directory has 
already been observed to cause conflicts with other applications (cvsweb), and 
this also means that we can never add a new filename without running the risk of 
yet another filename conflict.  With new features requiring new files being added 
to bugzilla all the time (I know of three or four files being added this next 
week for stuff being checked in for 2.14), this is an almost impossible 
assumption to make.  You're asking for trouble.

2) With Bugzilla's current directory structure, it is completely possible to have 
more than one Bugzilla installation on the same machine.  Most people who hack on 
Bugzilla (which is a LOT of the places that install it because they want to 
customize things) install a second one to use for development before subjecting 
their production version to premature hacks.  landfill.tequilarista.org is a REAL 
good example of this.  We currently have about 8 copies of Bugzilla running there 
and available from the web.  While technically possible to install a fully FHS-
compliant app in more than one place on the machine by providing the installer 
with an install prefix, you wind up with a messy directory structure again 
because you have needless levels of directory structure underneath the new 
install prefix.  Since you can have more than one install (and many people do) 
all of the files should be able to be grouped together so that it's easy to tell 
which installation they belong to.

3) currently it is possible for users to install a full working copy of bugzilla 
in their user web space (without needing admin cooperation, other than giving 
them access to the MySQL server, and granting ExecCGI privs).  A directory 
structure like the proposed would make life incredibly difficult for someone 
wanting to do it that way...

I have no objection to creating additional subdirectories within Bugzilla's tree 
to separate the types of files however...  I'm all for a clean directory 
structure.  I just hate to see the related files spread out all over the 
filesystem.

The cgi files and the static files should probably all still be in the same 
directory, if for nothing else than it makes a better aesthetic look to the URLs 
from the browser-standpoint.  To do anything else and still have the URLs look 
normal requires a lot of complicated rewrite rules in the web server config, and 
I don't want to require anyone to have to screw with rewrite rules.

I'd like to see default locations something like this:

  - html, js, gif, and cgi  -> /usr/share/bugzilla/web/
  - modules and .pl         -> /usr/share/bugzilla/lib/
  - localconfig             -> /usr/share/bugzilla/etc/
  - data/*                  -> /usr/share/bugzilla/data/
  - graph/*                 -> /usr/share/bugzilla/graph/

(shadow/* is purposely left out because it no longer exists)

Your web server would be pointed at /usr/share/bugzilla/web/

Of course, replace /usr/share/bugzilla with whatever you want if you have 
multiple installations, or just prefer it in a different place.  I have it in 
/usr/local/bugzilla on my servers.

On landfill, since we don't have access to edit httpd.conf, access.conf, or 
srm.conf, we'd be forced to put them in /var/www/${unique_install_name}/ and use 
.htaccess files to prevent people from getting into any directories other than 
web... (which is pretty much what we do now to keep people out of localconfig and 
data/*)

Comment 12

[Rémi Perrot]

I agree about the install of the cgi in /usr/lib/cgi-bin. I think I may have
misunderstanding  the Debian Policy or it's a bug of the Debian policy.
I agree about the fact that splitting installation across the file system make
development a nightmare, and having multiple installation a dream.
I don't care about default installation path if there is a way to change this in
a non interactive manner.

You don't have adapt bugzilla to Debian, this is my job. If you make bugzilla
closer to FHS this may help me but this has non sense for an install on Win32
for example.

Comment 13

[Dave Miller [:justdave]]

OK, cool...   Personally I happen to like FHS for most stuff...  Just from my use 
of Bugzilla it didn't seem appropriate for this particular situation.  Having a 
standard pattern for locating the files across platforms makes it easier to 
transition from one platform to another.  This is why I was thinking we should 
try to do something that would be compatible pretty much anywhere.

Perhaps classifying the files like that and allowing the configure script to 
override the base paths would be enough...

Comment 14

[Zach Lipton [:zach]]

I am cooking up an install program which Handles Stuff for installs. I am out 
of town (writing this from a B&B in Ashland Or.), but it will be done soon...

Comment 15

[Eric Hanson]

   Here's an idea I got out of 'Programming CGI with Perl' book.  I think it 
might make sense to take the localconfig file outside of the cgi directory.  We 
have everything but localconfig in:
/usr/local/apache/cgi/bugzilla

and then the localconfig file in:
/usr/local/apache/data/bugzilla

We could modify the current local config to read in a file we specify, a file in 
/usr/local/apache/data/bugzilla
Something like bugz_data_1 or whatever anyone wants.  
This would give us several advantages.
1.  You could have several bugzillas, and since each localconfig would go to a 
file we specify (the real localconfig data in teh apache/data/bugzilla directory) 
all the bugzillas would have thier own db, username, etc.
2.  We wouldn't have to care if the person running bugzilla can have a .htaccess 
file.  We can put all sensitive data into the apache/data/bugzilla directory.
3.  If the bugzilla admin doesn't have admin privilages on the the box, it 
doesn't matter.  If a person has a cgi directory in the home directory (set up by 
someone else):
/home/user/cgi/bugzilla
they would then setup a directory in:
/home/user/data/bugzilla

Hopefully this will make everyone happy.  Just an idea.

Comment 16

[David D. Kilzer (ddk)]

Attachment: Makefile.PL patch v1

Comment 17

[David D. Kilzer (ddk)]

I've written YAM (Yet Another Makefile) for Bugzilla.  I followed one
simple rule when doing this:  Since Bugzilla is written in Perl, a 
Makefile should be done the Perl Way.  Easier said than done, especially
after I realized that ExtUtils::MakeMaker was almost too limited to set
up my own groups of files to install.

My initial patch has been uploaded as Attachment 45265 [details] [diff] to this bug.  I'm
keeping my code in a CVS directory and synced with all commits, so 
future patches should apply easily against updated CVS directories.

Note that this patch also fixes Bug 44202 by using a built-in feature of
the ExtUtils::MakeMaker code.  (This is currently always done; a flag
could be added to turn this on and off.)


Conceptual changes:

  o New Makefile.PL for easily configuring and installing Bugzilla.
    Should be *easier* for Windows users to install Bugzilla this
    way.  Using it is just like a Perl module:

      perl Makefile.PL
      make
      make install
      make setup        # new for Bugzilla

    You may even write your own Bugzilla/Config.pm (or reuse an old
    one; or use your old 'localconfig' file by copying it into the
    directory) and run "perl Makefile.PL --defaults" instead!

  o Replace 'localconfig' with a real perl module named
    Bugzilla::Config (created from Bugzilla/Config.pm.in).  (If
    'localconfig' is copied into the directory, Makefile.PL will
    read from it if no Bugzilla/Config.pm file exists and convert
    it to the new format.)

  o Create groups of files (images, static HTML, CGI scripts, perl
    lib files, perl executables, documentation) for ExtUtils::MakeMaker
    and being able to install them anywhere (within reason) and still
    have Bugzilla function properly.

    This should make everyone happy as it will allow you to place
    every file in the same directory (as the current "installation"
    method does), or move all the files to FHS-ish locations as 
    desired.

  o Remove checksetup.pl by moving its functionality into Makefile.PL,
    setup.pl (database-agnostic setup stuff best handled by perl
    instead of a Makfile), and mysql/setup.pl (MySQL-specific setup).


Devilish details:

  o Every Perl script now has a 'use Bugzilla::Config;' line.  This
    needs to change to add a 'use lib ...;' before it so that
    multiple installations of Bugzilla may exist on one system
    (without doing something crazy like a chroot jail for each one).

  o All of the HTML files now have substitution variables in them
    for links to other HTML files/CGI scripts, which makes them
    potentially invalid HTML.  

  o The 'urlbase' parameter in defparams.pl was removed.  I'm not
    sure if this was very critical or not (its usage in various
    files was otherwise updated).  It could always be added back.

  o The version of Bugzilla moved from globals.pl to
    Bugzilla/Config.pm ($Bugzilla::VERSION).


Things that probably still need to be fixed:

  o Bugzilla/Config.pm is installed in a global perl lib directory.
    Needs to be moved to 'lib_dir', a 'use lib @@LIB_DIR@@;'
    needs to be added to all files that have 'use Bugzilla::Config;'
    in them, Bugzilla/Config.pm needs its 'use lib @@LIB_DIR@@;'
    line removed, and a variable replacement target needs to be added
    to MY::postamble() in Makefile.PL.  This will allow multiple
    installations on the same machine.

  o The 'config_dir' (usually /etc/bugzilla/) is not used.  Probably
    should have a flag to turn the use of that directory on and off.
    (Mostly there for Debian folks, of which I am a user. :^)

  o The robots.txt file should only be installed in the HTDOCS root
    directory.  Since we will support scenarios where a user is
    installing Bugzilla in thier ~/public_html/ directory, we need
    a flag to disable robots.txt being installed.

  o All the directory creation and permissions setting in setup.pl
    (formerly part of checksetup.pl) should be moved to targets in
    MY::postamble() in Makefile.PL to clean things up.

  o Where/how to install documentation in the ./docs/ directory?

  o Add a 'url_scheme' or 'use_ssl' config option to pick between
    'http' and 'https' protocols.

  o Add a config option to optionally add a perl executable path
    before any system() calls for Windows users.

  o installbin_generic() probably won't work for VMS.  (Anyone
    running Bugzilla on VMS?!)

  o I may have missed some links in some files.  I essentially
    grepped for every *.html, *.js, *.cgi, *.dtd and robots.txt
    filename in every file, but it's possible I still missed some.

  o See also: 'FIXME' comments in various files.


ExtUtils::MakeMaker (Makefile.PL) shortcomings:

  o prompt() needs to be more robust to handle different types of
    questions (string input, multiple choice, true/false, etc.)
    and defaults.  Better yet, ExtUtils::MakeMaker could use a
    perl module that does what the autoconf/configure scripts do
    for GNU software using shell utilities.

  o The ability to add "file groups" to WriteMakefile() parameters
    instead of the way I eval'ed hacked up MY::* override subroutines
    and manipulated the $self hashref to prevent warnings about my new
    variables.  A generic version of installbin() would be necessary
    to implement this (much like installbin_generic in Makefile.PL).

Comment 18

[David D. Kilzer (ddk)]

Attachment: perl-ExtUtils-MakeMaker-v5.45-CONFIGURE.diff (provides more info from 'CONFIGURE' param in 'Makefile' output)

Comment 19

[David D. Kilzer (ddk)]

Attachment 46454 [details] [diff] (perl-ExtUtils-MakeMaker-v5.45-CONFIGURE.diff) provides a
patch that you may use against the installed ExtUtils/MakeMaker.pm file
that came with your distribution of Perl.

IT IS FOR TESTING PURPOSES ONLY!

Basically, it prints out more information in the generated 'Makefile' about
what parameters are being passed back from the 'CONFIGURE' parameter in the
call to ExtUtils::MakeMaker() (in this case, the configure() subroutine).

Since this is used in Attachment 45265 [details] [diff], I thought you might find it useful
for testing.

FWIW, I've submitted this patch to the perl5-porters mailing list.

http://aspn.activestate.com/ASPN/Mail/Message/687027
http://aspn.activestate.com/ASPN/Mail/Message/687906

Comment 20

[David D. Kilzer (ddk)]

Attachment: Makefile.PL patch v2 (bugzilla-2.13-20010821-CVS-makefile.diff)

Comment 21

[David D. Kilzer (ddk)]

Attachment 46631 [details] [diff] brings the Makefile.PL patch up-to-date with changes in CVS
as of August 21, 2001 (12:00 AM PDT).  Other changes include:

  o Added back %urlbase% parameter as a read-only param.  Implementation of
    the read-only params is still a bit ugly (list of these params is in both
    globals.pl and defparams.pl).

  o Bugzilla/Config.pm is now installed in the 'lib_dir' config value, meaning
    multiple instances of Bugzilla may be installed on a single machine very
    easily.  A side-effect is that all perl files had to add a
    |use lib '@@LIB_DIR@@';| line that gets replaced when running 'make'.

  o Dropped 'config_dir' config value as it probably won't be useful to anyone
    other than Debian folks, and it's easy enough to add move one file into
    /etc and create a symlink during a build.

  o Now install stuff in ./docs/ under the 'docs_dir' config value.

  o Added 'url_schema' config value to set 'http' or 'https' for Bugzilla.
    (Needed after %urlbase% param was removed, then added back as read-only.)

  o Added 'perl_exe' config value for Windows users who must specify the
    path to perl (or perl.exe) when making calls using system().

  o Cleaned up all but 2 FIXMEs in Makefile.PL.

Stuff that could still be done (but probably isn't necessary):

  o The robots.txt file should only be installed in the HTDOCS root
    directory.  Since we will support scenarios where a user is
    installing Bugzilla in thier ~/public_html/ directory, we need
    a flag to disable robots.txt being installed.

  o All the directory creation and permissions setting in setup.pl
    (formerly part of checksetup.pl) should be moved to targets in
    MY::postamble() in Makefile.PL to clean things up.

Comment 22

[Zach Lipton [:zach]]

all your install are belongs to me!

Comment 23

[David D. Kilzer (ddk)]

Attachment: Makefile.PL patch v3 (bugzilla-2.14rc-20010827-CVS-makefile.diff)

Comment 24

[David D. Kilzer (ddk)]

Attachment 47232 [details] [diff] brings the Makefile.PL patch up-to-date with changes in CVS
as of August 27, 2001 (12:00 AM PDT), which is also known as version 2.14rc.

Also removed commented-out code from WriteMakefile() in Makefile.PL.

Comment 25

[David D. Kilzer (ddk)]

Attachment: Makefile.PL patch v4 - Bugzilla 2.14 (bugzilla-2.14-20010829-CVS-makefile.diff)

Comment 26

[David D. Kilzer (ddk)]

Attachment 47648 [details] [diff] brings the Makefile.PL patch up-to-date with the official
release of Bugzilla 2.14.

No other significant changes were made.

Comment 27

[David D. Kilzer (ddk)]

Attachment: Makefile.PL patch v4r2 - Bugzilla 2.14 (bugzilla-2.14-20010829-CVS-makefile-2.diff)

Comment 28

[David D. Kilzer (ddk)]

Attachment 47650 [details] [diff] fixes the Makefile.PL patch to work with the official
release of Bugzilla 2.14.  Really.  I actually tested it this time.
On the released tarball.  Previous patch had a small reject on globals.pl
'cause I didn't update my 'bugzilla-orig' tree to the correct version.

Comment 29

[David D. Kilzer (ddk)]

Attachment: Fix bug with perl_exe param for v2.14 patch (bugzilla-2.14-20010829-CVS-makefile-2-fix.diff)

Comment 30

[David D. Kilzer (ddk)]

Okay, for those of you playing at home, if you want to patch 
Bugzilla-2.14 to use the HIGHLY EXPERIMENTAL Makefile.PL patch, 
apply the following patches to a clean Bugzilla-2.14 tree:

1. Attachment 47650 [details] [diff] - bugzilla-2.14-20010829-CVS-makefile-2.diff
2. Attachment 47814 [details] [diff] - bugzilla-2.14-20010829-CVS-makefile-2-fix.diff

The "-fix" patch fixes a bug when setting the perl_exe parameter
for use with system() calls.

All future patch attachments will be against the CVS development 
version (v2.15).

Comment 31

[David D. Kilzer (ddk)]

Attachment: Makefile.PL patch v5 - Bugzilla 2.15 (bugzilla-2.15-20010908-CVS-makefile.diff)

Comment 32

[David D. Kilzer (ddk)]

Attachment 48851 [details] [diff] brings Makefile.PL patch up-to-date with CVS tip as
of Sep 08, 2001.  Incorporates use of Template::Toolkit changes, new
minimum Perl requirement and new required Perl modules.

Comment 33

[David D. Kilzer (ddk)]

At the suggestion of Matthew Barnson (Barnboy), I put up a "web site"
that will always have my latest version of the patch.

  http://debian.theracingworld.com/patches/bugzilla/

Note the date of the "LATEST" patch to determine what CVS snapshot
date you should use.  I will try to keep it updated daily, but I may
fall behind a bit over weekends or during busy periods at work, or
when major changes (like Template::Toolkit) are applied.

Comment 34

[Zach Lipton [:zach]]

*** Bug 44202 has been marked as a duplicate of this bug. ***

Comment 35

[John Stosh Muczynski]

Attachment: TODO for Windows NT One Step Install

Comment 36

[John Stosh Muczynski]

Attachment: First half of prototype for Windows NT One Step Install

Comment 37

[Zach Lipton [:zach]]

Spun off 104660 to cover the one-step install on NT, please keep this for 
unix issues and that for NT issues.

Comment 38

[Dave Miller [:justdave]]

I'll spell it bug 104660 so I can click on it :)

Comment 39

[Zach Lipton [:zach]]

ok, weeee! It is time for me to come up with something to say:

I used to think that I could ignore this bug and pretend I wasn't the 
installation owner and that all would go away! This isn't true. Basically, I 
like the idea of a makefile installation, but not quite the way that it is done 
here. IMHO, makemaker doesn't do what we need. Makemaker is a great 
tool for perl modules, but this isn't a perl module and I am not convinced 
that it is what we need as a distrobution. 

Why not do something like perl itself. Have a configure script which looks 
at various config info, tries to figure things out, and asks questions (but 
has a defaults mode for cronjobs to run non-interactively). Once the 
questions have been asked, it generates a Makefile and other config files. 
The Makefile then does the work of doing the install and would work as 
would be expected:

make -- makes and runs checksetup.pl
make test -- runs tests
make install -- installs

We would also have make upgrade which would handle a checkout of the 
latest verson (tip of version) from cvs and the upgrade process. This 
could come later though.

Most of the work already done could stand, all the stuff that is done with 
the config vars looks good. But getting that information is probably better 
done a different way.

First of all, we need to live on a branch. The patch thing is getting insane. I 
will get us a branch and handle commits to it ASAP. See bug 105848.

I also have a configure template which uses a nifty (if I may say so myself) 
ConfigureModule system. Assuming I'm not going insane, I'll post this 
soon as well.

Please let me know if I have gone off the deep end here, if not, I will open 
up other bugs to cover specific areas for this and turn this into a meta 
bug.

Comment 40

[Zach Lipton [:zach]]

this is now a meta bug

Comment 41

[Zach Lipton [:zach]]

the bugzilla_newinstall branch has been created, after a quick reality 
check (not a review) with someone else, please check in all patches 
there. I'm not putting the main patch in yet since it will need some 
refacoring.

Comment 42

[Zach Lipton [:zach]]

the bugzilla_newinstall branch has been created, after a quick reality 
check (not a review) with someone else, please check in all patches 
there. I'm not putting the main patch in yet since it will need some 
refacoring.

Comment 43

[Zach Lipton [:zach]]

Pushing to the 2.18 timeframe

Comment 44

[Zach Lipton [:zach]]

Comment on attachment 48851 [details] [diff] [review]
Makefile.PL patch v5 - Bugzilla 2.15 (bugzilla-2.15-20010908-CVS-makefile.diff)

Patch is way too stale, but development is occuring on the bugzilla_newinstall
branch

Comment 45

[Diego Biurrun]

*** Bug 173598 has been marked as a duplicate of this bug. ***

Comment 46

[John Stosh Muczynski]

Comment on attachment 53437 [details]
First half of prototype for Windows NT One Step Install

see bug 104660

Comment 47

[Dave Miller [:justdave]]

*** Bug 221581 has been marked as a duplicate of this bug. ***

Comment 48

[Dave Miller [:justdave]]

Bugzilla 2.20 feature set is now frozen as of 15 Sept 2004.  Anything flagged
enhancement that hasn't already landed is being pushed out.  If this bug is
otherwise ready to land, we'll handle it on a case-by-case basis, please set the
blocking2.20 flag to '?' if you think it qualifies.

Comment 49

[Frédéric Buclin]

Max, we have a well working checksetup.pl. Do this bug/request and its blockers still make sense? I would tend to won't fix them.

Comment 50

[Max Kanat-Alexander]

Right, we're not going down this path. We're definitely not using a Makefile installer.