799637 - AnnotatedSummaryGenerator.php's BzAPI orange bug search should specify the exact columns required

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Ed Morley [:emorley]]

If columns aren't specified when making a bzapi request, you get the default 150 fields, which requires more joins on the b.m.o DB side & means a larger json response.

At most we need:
{bug number, summary, status, resolution, last changed}

(and in some places, less)

Need to change:
* https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/php/inc/AnnotatedSummaryGenerator.php#l100
* https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/js/Data.js#l28

(And if bug 799603 were to end up being wontfix:
* https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/9c7f496beb97/php/submitBugzillaComment.php#l32 )

Comment 1

[Ed Morley [:emorley]]

Attachment: WIP

The AnnotatedSummaryGenerator.php change has uncovered a few BzAPI oddities when using include_fields with 'resolution', as such I've broken the UI part (link #2 in comment 0) out to bug 800875, so it's not blocked on them.

Link #3 in comment 0 is no longer relevant, as that block has been removed.

This just leaves the AnnotatedSummaryGenerator.php change, for which I'm check-pointing a WIP patch.

It:
* Adds "&include_fields=id,summary,status,resolution"
* Removes the (I believe to be) unnecessary mapping in the array_map function - which I can only think was added to strip out the unwanted fields, to avoid sticking them all in the bugscache.
* Adds the resolution field if we didn't get it, to work around bug 800646.

TODO:
* Pending the result of bug 800652, add our own sorting of the bugs array, so the bug suggestions shown in the annotated summary are in the same order as they are currently (or at least have open bugs at the top).

--

In the future, I want to replace the bugscache with a proper cache of all filed orange bugs, which will make this bzapi call redundant. Combined with my desire for us to eventually split out closed bug suggestions in the annotated summary (and put under a "show more", which means the sort order won't matter); it's quite a pain this wasn't just a simple case of adding include_fields as a stop-gap - since we'll be replacing much of this soon :-(

Comment 2

[Ed Morley [:emorley]]

Oh and I forgot the other fun part:

Since we no longer time out as much in the bzapi call, we now return ridiculous number of bug suggestions for certain failure types (that we previously wouldn't have given any suggestions for).

eg "Shutdown"

https://api-dev.bugzilla.mozilla.org/latest/bug?whiteboard=orange&include_fields=id,summary,status,resolution&summary=Shutdown

So I'll need to figure out what else to add to the ignore filter.

Comment 3

[Ed Morley [:emorley]]

Attachment: Patch v1

* Now that bug 800646 and bug 800652 are fixed, the returned JSON no longer differs when include_fields is used, so the workarounds mentioned previously are not needed.
* Now that we don't get a load of unused fields, we don't need to do the array_map() dance to filter them out :-)

Comment 4

[Ed Morley [:emorley]]

https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/43b5054776ba

Comment 5

[Ed Morley [:emorley]]

Looking at tbpl-dev import logs, this appears to have reduced bug lookup times from ~4400-4700ms to ~2300-2700ms - pretty happy with this :-)