6419 - want command-line queries for bugzilla

Status: RESOLVED FIXED

(Bugzilla :: Query/Bug List, enhancement, P2)

Description

[Akkana Peck]

It would be very cool to have some command-line queries for bugzilla, like SGI's
pvquery.

I don't remember the exact syntax of pvquery, but it had flags for the various
options you might want to pass in in a bug query, for example (I'm making up
flag letters here)

pvquery -Aakkana -P1,2 -snew,assigned -Rkostello -MM1,M2,M3,M4,M5,M6 -Sclipboard

might get me all the bugs assigned to akkana, priority 1 or 2, status new or
assigned, reported by kostello, milestones1 through 6, with "clipboard" in the
summary (note I picked s for status and S for summary; alternately maybe S for
summary and arguments without a flag match stuff in the summary?)

The main thought is that a command-line query like this might be faster than
going through a browser, and it would be easier to redirect output without
dealing with 4.x's annoying reformatting when converting copied text to
plaintext (though hopefully mozilla will soon be to the point where we don't
need to use 4.x any more :-)

Comment 1

[Andrew Taylor]

Command line bug submission would be cool, too.  Then it would be possible to do 
stuff like email bug submission.

Comment 2

[Terry Weissman]

[e-mail] is the new owner of Bugzilla and Bonsai.  (For details,
see my posting in netscape.public.mozilla.webtools,
news://news.mozilla.org/38F5D90D.F40E8C1A%40geocast.com .)

Comment 3

[Tara Hernandez]

Brian--have at it.  Prototype something up and let's see what we get.  Probably
worth working with akkana as you go.

Comment 4

[Dave Miller [:justdave]]

Something to point out is that with bug_email.pl as far along as it is now (we 
can already submit via email and add comments), this would make queries via email 
easy to implement, too.

Comment 5

[Brian Ostrom]

Sounds like fun.  I'll start playing with this when I have time.
And it'll be Unix-only, of course.... ;-)

Comment 6

[Brian Ostrom]

Okay.  I started playing around with a couple of prototype approaches.
The first (of course ;-) was a shell script, which worked fine, but
requires rsh, so it can't be used in anything but an intranet setting.
So then I wrote a C program with the vague hope that mysqld might be
doing something sane with port 3306 (hopefully this is the default).
It opens a socket, does a connect, a send, and a recv.  But all I get
back is "3.22.19<and random gibberish>".  Oh, I forgot to mention that
in order to get even this far, I had to enable 'select' access from
any user on any host (*just* select, so in theory no one could do
anything malicious).

I guess it's time to grab the mysql source, and try to figure out what
I'm doing wrong.

Note: I'm trying to avoid using Perl, because I want this to be small,
fast, and not reliant on having Perl and the required obscure modules
installed.

Comment 7

[Ben Bucksch (:BenB)]

I filed bug 31907 for using NNTP for comments.

If you could at least make a reply to the bugzilla spam work, that would totally
rock.

Comment 8

[Dave Miller [:justdave]]

I think you got the wrong bug number...  that one is something to do with 
preferences in Mail/News in Mozilla.

Replying to the bugzilla spam to add comments is already possible (but doesn't 
really have much to do with this particular bug).  Look at bug_email.pl and 
appendbugmail.pl (or whatever their real names are) and the associated README in 
the contrib/ directory.

Comment 9

[Ben Bucksch (:BenB)]

Sorry, the right bug is 31706.

Dave,
cool!

However, I am no admin, just a Mozilla bugzilla user, so a perl script doesn't
help me at all. Can't we set this up for Mozilla's bugzilla?

Comment 10

[Dave Miller [:justdave]]

Ah, that's right...  mozilla.org doesn't use it.  I think this is because the 
error messages and confirmation notices it sends out still leave a lot to be 
desired, and they'd tend to scare people off.  But this is a topic for another 
bug report, it's off topic here.  (I just looked, and I don't see one anywhere..  
if I get around to it I'll open a new one for that later tonight).

Comment 11

[dror2]

The simplest hack for command-line interface to bugzilla would be a lynx-based 
script (That is, actually submitting requests through the web interface).
It has the property of being independent and works over the Internet.
However, it has the drawback of being limited to the current HTML format of web 
pages... (that is, it would break on every GUI change..)

Comment 12

[Akkana Peck]

The advantage of command-line queries is that they could be noninteractive --
e.g. I could write a script to have bugzilla mail me my buglist at particular
times, or (if it extends to changes, not just queries) to close a bug at the
same time as I'm checking in a fix.  Just making it non-GUI using lynx wouldn't
help much -- sure, you could write a script to send characters to lynx, but that
wouldn't really be any easier than writing a script to send queries to bugzilla
over an http pipe.

Comment 13

[Andreas Franke (gone)]

What about turning http://www.ags.uni-sb.de/~afranke/mozilla/quicksearch.html
into a server-side CGI script and using wget ?
Akkana's original query would have to be translated to

"assignedto:akkana priority:1,2 status:new,assigned reporter:kostello
milestone:m1,m2,m3,m4,m5,m6 summary:clipboard"

or shorter

"NEW,ASSI @akkana p1,2 rep:kostello tm:m1,m2,m3,m4,m5,m6 +clipboard"
 
The only problem is that m1 is treated as a substring like everything else,
therefore m10,...m19 bugs would show up. But anyway, who wants to query for m1
bugs now? Or, if this is a serious problem, why not enhance that tool with a
generic "exact match" shortcut? Or we just special case the milestone field and
use exact matching instead of substring matching there.

Comment 14

[Andreas Franke (gone)]

Ah, and just in case it helps anybody, here is a shell script to extract the bug
numbers from a buglist.cgi generated html page:
#!/bin/sh
## usage: bugids <inputfile>
echo `grep 'TR VALIGN=TOP ALIGN=LEFT CLASS=' $1 | sed -e 's/<TR.*id=//' | sed -e
's/".*//'` | sed -e 's/ /\,/g'

Comment 15

[Andreas Franke (gone)]

Attachment: query.conf

Comment 16

[Andreas Franke (gone)]

Attachment: buglist*

Comment 17

[Andreas Franke (gone)]

Attachment: bugs*

Comment 18

[Andreas Franke (gone)]

I have attached a fix for this bug. It's not perfect, but it seems to work.

The query.conf file contains the mapping from options to field names and
comparison types. Quoted option names are `grep'ped for, so it should be easy to
edit this file. Comments (#) have no effect; you have to make sure that these
lines do not contain any quoted "option".

buglist is a tcsh script that submits a bugzilla query and writes the resulting
html page to stdout. It supports both short options ("-Afoo", "-Rbar") and long
optiones ("--assignedto=foo", "--reporter=bar"). If the first character of an
option is not "-", it is treated as if was prefixed with "--default=".

The columnlist for the result is currently hardcoded, but it sould be easy to
make it use the same setting that's in ~/.netscape/cookies . The following
commands may help:
set line=`grep COLUMNLIST ~/.netscape/cookies`
set start=`expr match "${line}" '.*COLUMNLIST'`
set start=`expr ${start} + 1`
set len=`expr length "${line}"`
set len=`expr ${len} - ${start}`
set columnlist=`expr substr "${line}" ${start} ${len}`
For mozilla, I'm not sure how to find out which profile to use.

bugs is a simple tcsh script that calls buglist and then extracts the bug
numbers from the output. Adding the prefix 
	http://bugzilla.mozilla.org/buglist.cgi?bug_id=
turns the bug list into a working link (if any bugs were found at all).
Counting them is easy: Just pipe the result through this:
	sed -e 's/,/ /g' | wc | awk '{printf $2 "\n"}'

Akkana, are you still interested in this?

Comment 19

[Stephan Niemz]

Not everybody has tcsh installed, so I'm not sure if this helps that much.

Besides, csh programming is considered harmful, see
http://www.landfield.com/faqs/unix-faq/shell/csh-whynot/ for the reasons.

IMHO the best would be a Perl or maybe a shell script.

Comment 20

[Andreas Franke (gone)]

Attachment: buglist* for sh

Comment 21

[Andreas Franke (gone)]

Attachment: bugs* for sh

Comment 22

[Andreas Franke (gone)]

faniz: Thank's for the link about csh. BTW it's easy to criticize others. I know
that you can program in perl. What about porting the make_query_URL function
from http://bugzilla.mozilla.org/showattachment.cgi?attach_id=26161 (bug 69793)
to perl then?

Comment 23

[Akkana Peck]

Sounds awesome!  I don't care whether it's tcsh or sh or perl or some other
language -- that page ranting about csh is pretty silly, every language has some
disadvantages and I could easily write a page ranting about problems in sh or
perl.  Most people on Unix these days have tcsh available.  (Note, your
attachment for "bugs* for sh" begins with "#!/bin/tcsh -f", might be a typo.)

Both the tcsh and sh versions call the command wget, which I don't have.  Is
that a separate package I need to install.  (The rpmfind server is down, so I
couldn't search there.)

Comment 24

[Andreas Franke (gone)]

Attachment: bugs* for sh (for real this time)

Comment 25

[Andreas Franke (gone)]

http://rpmfind.net/linux/rpm2html/search.php?query=wget seems to be up now.

Comment 26

[Akkana Peck]

Oh, that's very cool.  I customized the columns for my screen width and piped
buglist through w3m -T text/html -dump and the result looks great!  And I can
pipe to wc and grep and other helpful utilities.

Looking at the Netscape cookies to decide which columns to show sounds cool, but
actually I like it better this way (or else customizable through command-line
options), since I'll probably want to show fewer columns in my 80-column text
windows than I do in a browser window that uses a proportional font.

I'll definitely use this, and bet other people will like it, too -- Thanks!

We should check this in to the tree somewhere (mozilla/tools ?) and advertise
it.  If we think any win people might want this (not sure how likely that is), I
could translate it into perl, but for Unix people it's fine in either shell
language.

Comment 27

[Stephan Niemz]

Andreas: sorry, I think my comment was a bit too harsh.  I just find *csh
scripts very bad to maintain.

I'm already planning on writing a script to attach files to a bug directly from
the command line.  If I find the time, I could indeed port your quicksearch to
perl.

Comment 28

[Andreas Franke (gone)]

Attachment: patch for the sh version of buglist to support $COLUMNLIST

Comment 29

[Andreas Franke (gone)]

Attachment: buglist* (new sh version) with support for $COLUMNLIST

Comment 30

[Andreas Franke (gone)]

I think the COLUMNLIST problem is solved now: it's not a commandline option, but
an environment variable. The 'env' program makes it easy to use though:

env COLUMNLIST="priority summaryfull" buglist --status=NEW --assignedto=akkana

Comment 31

[Matthew Tuck [:CodeMachine]]

Consider for 2.16.

Comment 32

[Christopher Aillon (sabbatical, not receiving bugmail)]

-->New product Bugzilla

Comment 33

[Zach Lipton [:zach]]

Returning to default owner.

Comment 34

[Gervase Markham [:gerv]]

Is there any reason we can't just check this script (with some documentation,
Andreas ;-) into contrib?

Gerv

Comment 35

[Andreas Franke (gone)]

Documentation is already available here:
http://www.trilobyte.net/barnsons/html/cmdline.html

IMO the contrib directory would be a good place :-)

This tool consists three files:
query.conf	attachment 26157 [details]
buglist		attachment 26944 [details] [diff] [review]
bugs		attachment 26215 [details] [diff] [review]
where "buglist" and "bugs" must be executable.

Comment 36

[Gervase Markham [:gerv]]

I'd like to check these in, but they don't have appropriate license headers.
Andreas (or anyone who knows what is position is): I assume that these files are
licensed under the standard Bugzilla license (the MPL)?

Gerv

Comment 37

[Andreas Franke (gone)]

Gerv: feel free to put in any license you want.

Comment 38

[Gervase Markham [:gerv]]

Checked into contrib/cmdline/ .

Gerv

Comment 39

[Maciej Maczynski]

I was using buglist script happily for a long time on linux.
Now I've moved to solaris 7 and the script does not work any more.
It looks like solaris version of sh does not like "length" and "substr".
Also "let" and "==" in 'if' are not wellcome.

I've made the script work with the following changes:
- using /usr/ucb/expr instead of /usr/bin/expr 
 (set the variable expr=/usr/ucb/expr and use ${expr})
- changing "==" to "="
- changing every "let a=b+c" to "a=`${expr} ${b} + ${c}`"

I can submit a patch, but I'd like an opinion from someone who is
more competent in shell scripts then I.  I mean is there a way to use 
the original script on Sun without the changes I've made?

Comment 40

[Andreas Franke (gone)]

- using /usr/ucb/expr instead of /usr/bin/expr 
 (set the variable expr=/usr/ucb/expr and use ${expr})

I think there are several different approaches:
* the lazy one: rely on some tools to be in your $PATH, document the required
versions with comments in the script file, and leave it up to the admin who
installs the script to ensure that it works with the default $PATH or that the
script is changed to set it's own $PATH appropriately.
* store the path to each tool in a variable, and refer to the variable each time
you are calling the script (like you suggest with ${expr}). If the path is
different on the installation site, the admin just needs to tweak a few lines at
the top of the script.
* autodetect the appropriate tools with configure at install time and let
configure replace the placeholders in the script.in file when it generates the
script

IMO, the configuration approach is too complex here. The use of ${tool} for
every call makes scripts slightly less readable in my experience, though I have
often used it in some of my scripts (and the "buglist" script is can hardly be
called readable anyway). When I was writing the script, I was hoping to get
around by only using features that are supported by all commonly used toolsets.
I don't know what version of expr (and if it's gnu or something else)
you have in your /usr/bin directory, and why it is in your $PATH before /usr/ucs
, but if it exists and changing the default $PATH globally is not an option,
then your ${expr} solution sounds fine, to me at least.
BTW, I have the same problem on our solaris 2.6 machines, but we are using
/bin/expr , and I can't get that one to print a version...

- changing "==" to "="

I think this should work on Linux, too. (At least it worked when I tried.) 
So this change should be fine.

- changing every "let a=b+c" to "a=`${expr} ${b} + ${c}`"

I don't know enough about let to make a call here, but it makes sense to me.
One less dependencies on shell builtins. If you get around using ${expr} in the
first issue, then this could be "a=`expr ${b} + ${c}`", of course.

Note that I'm not an expert in shell programming. If someone has real knowledge
in this area, please share.

Comment 41

[Andreas Franke (gone)]

In bug 278829, there is a new, Bugzilla 2.18 compatible version which might
solve the solaris problems:  it doesn't use 'let' or '==', only `expr foo :
someregexp` matching which seems to work on both linux and solaris.

If there are still issues, please open new bugs for them and make them block bug
278800.