721366 - Release Mediawiki-Bugzilla on Mozilla Wiki

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Brandon Savage [:brandon]]

There is an extension for mediawiki and the wiki.moz installation that needs to be installed on the server.

The code is available in Github (https://github.com/brandonsavage/mediawiki-bugzilla) and has been through secreview. 

Please ping me for additional details you may need.

Comment 1

[Dumitru Gherman [:dumitru]]

Brandon, can you link here the bug where infrasec reviewed the extension? Thanks!

Comment 2

[Laura Thomson :laura]

Sec review bug is resolved fixed https://bugzilla.mozilla.org/show_bug.cgi?id=706936

Comment 3

[Jake Maul [:jakem]]

This seems somewhat non-trivial (has prerequisites). I'm going to do it on wiki.allizom.org first to make sure it doesn't break anything when we first turn it on.

Comment 4

[Jake Maul [:jakem]]

Attempted to deploy on wiki.allizom.org, could not create the bugzilla_cache table:

Creating bugzilla_cache table...A database query syntax error has occurred.
The last attempted database query was:
"CREATE TABLE IF NOT EXISTS `bugzilla_cache` (
 `id` integer(40) NOT NULL AUTO_INCREMENT,
 `key` varchar(255) NOT NULL DEFAULT '',
 `fetched_at` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
 `data` longtext,
 `expires` integer(11) NOT NULL DEFAULT 0
 PRIMARY KEY (`id`)
 ) ENGINE=InnoDB, DEFAULT CHARSET=binary
"
from within function "DatabaseBase::sourceFile( /data/www/wiki.allizom.org/extensions/Bugzilla/cache.sql )".
Database returned error "1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(`id`)
 ) ENGINE=InnoDB, DEFAULT CHARSET=binary' at line 7 (10.8.70.94)"


Please let me know when I can git pull and try again. Thanks!

Comment 5

[Brandon Savage [:brandon]]

Not sure why I didn't catch this when I tested the upgrade but never the less it is now fixed in master.

Comment 6

[Jake Maul [:jakem]]

This is installed on wiki.allizom.org and sorta working- the dependencies aren't there yet. I need to figure out we want to do those in puppet so it's repeatable.


http://www.smarty.net/download

Possibly we can use this these?
http://rpmfind.net//linux/RPM/epel/5/x86_64/php-Smarty-2.6.26-1.el5.noarch.html
http://rpmfind.net//linux/RPM/epel/beta/6/ppc64/php-Smarty-2.6.26-1.el6.noarch.html

This installed fine on the staging server, so hopefully it'll work as-is. Can't really test by itself, basic usage seems to be reliant on HTTP_Request2.



I've tried HTTP_Request2 from PEAR (which puppet should support), doesn't work... the version of php-pear on the staging box is too old (stock RHEL5 version):

pear/HTTP_Request2 requires PEAR Installer (version >= 1.5.4), installed version is 1.4.9

It looks like the prod cluster (pm-app-generic*) has a newer one... simplest solution might be to get the rpm from prod over to mrapp-stage04, and install this from PEAR. Puppet should be able to manage that.

Comment 7

[Brandon Savage [:brandon]]

Smarty is in fact no longer a dependency. I recommend upgrading the version of the PEAR installer you're using, and trying that way. Does Puppet have root access? Most of the time, those files are written to areas of the filesystem that require elevated permissions.

Comment 8

[Jake Maul [:jakem]]

I've gotten this to work on staging, using the RPM for php-pear from prod and 'pear install HTTP_Request2'. That should be doable in puppet.

I had to increase Mediawiki's memory_limit from 64MB to 128MB.

I used a simple version of the query on the github page:
<bugzilla>
    {
        "product": "mozilla.org",
        "component":"Server Operations: Web Operations"
    }
</bugzilla>


This returns *all* bugs in that product/component, regardless of status... hence the memory usage.

I'm afraid we're going to run into massive memory usage problems due to situations like this (users forming bad queries). What can we do to mitigate this?

For starters, we should have some examples of good queries, so people get can copy and modify them to suit. I don't know the bugzilla API well, and I don't think most of wikimo's users will either. In this case, what would the syntax be to select only bugs that are unconfirmed,new,assigned,reopened (aka: open bugs)? After some reading, I know how it would work in a simple GET request, but not how to specify it in this form (the docs this links to give URL examples, not JSON examples).

However, I'm not sure this is sufficient by itself. Can we reduce or limit the memory usage of these queries somehow? Obviously we can cap it with memory_limit, but that's rather mean as it results in a blank page and some difficulty getting back to the edit page (can't just click "Edit" as the page is blank... gotta know the URL format or be able to go back). It'd be very non-obvious if a working page suddenly stops working because of memory_limit.

Can we do something like capping the number of results from Bugzilla? I don't know if the API has anything like that, or if there's a better way altogether. Maybe the size of the return is okay but the data structure used by this code has a lot of overhead...

Comment 9

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Jake Maul [:jakem] from comment #8)
> I'm afraid we're going to run into massive memory usage problems due to
> situations like this (users forming bad queries). What can we do to mitigate
> this?
>
I think many of the queries will return relatively small (>50) bug lists - at least initially. At what size result set do we get into memory issues? Are you concerned that the server is going to be overwhelmed by requests? Does this have the potential of taking down wikimo?

> For starters, we should have some examples of good queries, so people get
> can copy and modify them to suit. I don't know the bugzilla API well, and I
> don't think most of wikimo's users will either. In this case, what would the
> syntax be to select only bugs that are unconfirmed,new,assigned,reopened
> (aka: open bugs)? After some reading, I know how it would work in a simple
> GET request, but not how to specify it in this form (the docs this links to
> give URL examples, not JSON examples).
>
This is a good idea. We should create a doc with some samples to show people how to use this new feature if this has not already been done. 

Brandon - Have either Christian or you documented the feature somewhere?
 
> However, I'm not sure this is sufficient by itself. Can we reduce or limit
> the memory usage of these queries somehow? Obviously we can cap it with
> memory_limit, but that's rather mean as it results in a blank page and some
> difficulty getting back to the edit page (can't just click "Edit" as the
> page is blank... gotta know the URL format or be able to go back). It'd be
> very non-obvious if a working page suddenly stops working because of
> memory_limit.
> 
I'm not too concerned about the blank page in the beginning as no one is yet using this feature. We can tell people about the result set limits and the resulting issue up front.

> Can we do something like capping the number of results from Bugzilla? I
> don't know if the API has anything like that, or if there's a better way
> altogether. Maybe the size of the return is okay but the data structure used
> by this code has a lot of overhead...

There are a number of approaches that we can take. 
1. We can look at reducing the memory usage by the current implementation. 
2. We can implement pagination. 
3. We can cap the result set and provide a link to the full result set if it exceeds a certain threshold.

Unless there is a concern about this feature crashing wikimo I think that we should go ahead and deploy with the known limitation (need to know the result set size that causes the issue). We will document the issue and can work on a fix for it in the coming weeks.

Comment 10

[Jake Maul [:jakem]]

On the documentation: I'd be interested to know some of the following:

How to specify multiple elements in a field (ex: multiple status values)
which columns are visible
sorting


I'm not concerned about breaking wikimo... the plugin doesn't really cause any load on wikimo that I'm aware of, but rather just stalls and waits for Bugzilla.

My concern is that if used improperly it can break individual pages, to the point of being difficult to fix them. The page doesn't load at all, so you can't just click the 'edit' button to fix it... you need to know how to mangle the URL to get to the edit page.

As far as the size issue, I wouldn't be so sure it's not a problem even right away. The very first query I created returned 4k+ results, simply because I didn't know enough to limit it properly.

All of those sound like good/feasible ideas to reducing memory usage.


I'm okay with taking this live once we can set proper expectations and give good example queries. if we tell people "big queries won't work" and they make big queries anyway, I can live with that.

Let me know when we have some more example queries that people can build from... would love to push this live soon. :)

Comment 11

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Jake Maul [:jakem] from comment #10)
> How to specify multiple elements in a field (ex: multiple status values)
> which columns are visible
> sorting
> 
From playing around it's not clear to me that this is currently supported...but it really should be. 

Brandon - Is there a way to specify multiple values for a field? Multiple status values is a very good example of where this will frequently need to be done.

> My concern is that if used improperly it can break individual pages, to the
> point of being difficult to fix them. The page doesn't load at all, so you
> can't just click the 'edit' button to fix it... you need to know how to
> mangle the URL to get to the edit page.
> 
It look to me like the magic URL scheme is as follows.
If I'm on a page like:
https://wiki.mozilla.org/Program_Management/Programs/Silent_Update
In order to edit this page I will use the URL
https://wiki.mozilla.org/index.php?title=Program_Management/Programs/Silent_Update&action=edit

More generally:
https://wiki.mozilla.org/PagePath/PageTitle
https://wiki.mozilla.org/index.php?title=PagePath/PageTitle&action=edit

Correct?

Comment 12

[Brandon Savage [:brandon]]

There is some documentation on usage in the wiki, but it's likely incomplete and could stand to be expanded. Lawrence, did you try comma-separated values for multiple status values?

Comment 13

[Brandon Savage [:brandon]]

The bug comment that contains Christian's original spec is https://bugzilla.mozilla.org/show_bug.cgi?id=691829#c2 and this is what I worked from when I updated his plugin. We can certainly add to it, as we need.

Comment 14

[Lawrence Mandel [:lmandel] (use needinfo)]

(In reply to Brandon Savage [:brandon] from comment #12)
> There is some documentation on usage in the wiki, but it's likely incomplete
> and could stand to be expanded. 

Do you have a pointer to the documentation? If we want this to be applicable to other mediawiki installations, should we keep the documentation in the readme file on github?

>Lawrence, did you try comma-separated values
> for multiple status values?

I did try comma separated values. From what I can tell, this will result in a query being passed to bugzilla with a param like "status=unconfirmed,new" where the param should be passed as "status=unconfirmed&status=new". I think the existing code can be easily modified to take a comma separated list and create a separate param for each entry as required by the bugzilla api.

Comment 15

[Lawrence Mandel [:lmandel] (use needinfo)]

I have updated the readme doc with more query information and known issues including how to recover from a blank page. You can see the changes at 
https://github.com/lmandel/mediawiki-bugzilla/blob/master/README.md
Once we're happy with the scope of the changes I'll initiate a pull request.

I also noticed that the bar chart fails with the following error:

Warning: imagepng() [function.imagepng]: Unable to open '/data/www/wiki.allizom.org/extensions/Bugzilla/charts/b52ecfb625dc6e2c38384d8c4f076225.png' for writing: Permission denied in /data/www/wiki.allizom.org/extensions/Bugzilla/pchart/class/pImage.class.php on line 151

Is this intentional?

Comment 16

[Brandon Savage [:brandon]]

That error is not intentional. It occurs because the charts directory is not writable by the webserver. In order to avoid the heavy lifting of processing the charts every time we load the page, we cache them for a period of time on the file system; this seemed the most prudent method.

Comment 17

[Michael Coates [:mcoates] (acct no longer active)]

Moving back to server ops - I don't think security is needed. Please update if I'm incorrect.

Comment 18

[Lawrence Mandel [:lmandel] (use needinfo)]

Looks like I made the component change. That was accidental. Sorry for the churn.

(In reply to Brandon Savage [:brandon] from comment #16)
> That error is not intentional. It occurs because the charts directory is not
> writable by the webserver. In order to avoid the heavy lifting of processing
> the charts every time we load the page, we cache them for a period of time
> on the file system; this seemed the most prudent method.

Yeah. My question was really targeted to Jake. Didn't know if there was a reason why this would be disabled on the staging environment.

Comment 19

[Lawrence Mandel [:lmandel] (use needinfo)]

Jake - Is there any outstanding information that you need before deploying? Can you get mediawiki-bugzilla deployed early this week?

Comment 20

[Jake Maul [:jakem]]

On the charting issue: is this a configurable path (and/or could we make it one)? It would be nice to be able to put these in /tmp instead of inside the DocumentRoot, if we can. I can symlink it of course but it might be cleaner as a config setting... this way you could remove it entirely from the git repo (btw, there is a charts/.DS_Store file in git that should be removed).

wiki.mozilla.org is not served by a single server- the charts/ directory is not shared between them, meaning the caching is not nearly as effective as one might think- a chart generated on one web node won't be visible to another. That's another reason why I'm wondering about a config setting... instead of being in the DocRoot or pointing at /tmp, it could point at a shared NFS mount. A symlink would still work here, too.

The changes to the README.md in comment 15 is very informative. With that I'm content to push this to prod from a documentation standpoint. My only concern at this point is how to handle the chart caching.

Comment 21

[Brandon Savage [:brandon]]

(In reply to Jake Maul [:jakem] from comment #20)
> On the charting issue: is this a configurable path (and/or could we make it
> one)? 

Yes. See lines 149 - 151 of Bugzilla.php. You should be able to configure $wgBugzillaChartStorage and you'll need to properly configure the $wgBugzillaChartUrl URL as well.

Comment 22

[Lawrence Mandel [:lmandel] (use needinfo)]

If charting is already configurable, great. If not, I think it's OK if we should go ahead without functioning charts at this point. I've opened bugs for:
1. paging the results (comment 9) - bug 729255
2. supporting the use of multiple parameters with the same name, aka status: new, status: unconfirmed (comment 10) - bug 729259
3. making the chart path configurable (comment 20) - bug 729263 - submitted before I saw Brandon's reply.

Comment 23

[Jake Maul [:jakem]]

This is deployed and operational!

Simple test of both table and chart:
https://wiki.mozilla.org/User:Nmaul


In the end I opted for a symlink... this felt a bit more straightforward and obvious than changing the config file plus adding an Apache alias to make the URL work... I hadn't considered the latter.

How about a forum/yammer/blog post about this with a link to the README on github? :)

Comment 24

[Jake Maul [:jakem]]

Note that due to our deployment model, this code needs to cloned in one location, then copied (minus .git and .gitignore) into the extensions directory on mradm02... so for the time being updating is not a simple 'git pull'. If you need to update, please reference this bug or the proper git repo to pull from, so we can find it easily. :)

The problem is just that our deployment method uses git, and is highly confused by having .git trees within itself. This will get better when this site moves to PHX1 on a new cluster (Q1/Q2 timeframe), which explicitly handles issues like this.