1035804 - BzAPI compatibility layer bug query response takes 76s vs 6s for rest

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Extensions, defect)

Description

[Ed Morley [:emorley]]

Found whilst deploying bug 1034524.

https://api-dev.bugzilla.mozilla.org/latest/bug?keywords=intermittent-failure&include_fields=id,summary,status,whiteboard&changed_after=2014-01-09
-> GET https://api-dev.bugzilla.mozilla.org/latest/bug [HTTP/1.1 200 OK 6520ms]

https://bugzilla.mozilla.org/bzapi/bug?keywords=intermittent-failure&include_fields=id,summary,status,whiteboard&changed_after=2014-01-09
-> GET https://bugzilla.mozilla.org/bzapi/bug [HTTP/1.1 200 OK 76603ms]

Comment 1

[:glob ✱]

the search itself is fast.

time is spent in two places:
1. creating bug objects for each bug in Bugzilla::WebService::Bug::search() .. new_from_list()
2. search_bugs_response (morphing each bug)

Comment 2

[David Lawrence [:dkl]]

It is not a trivial fix as I originally though as Bugzilla::Search does not return product_id and component_id in the results data. So it is not as simple to take the results data, convert to hash
and pass to Bugzilla::Bug->new_from_hash. Bugzilla::Search would need to be altered as well to support
this.

dkl

Comment 3

[:glob ✱]

on my development system that query takes ~ 21s
i committed bug 880097 last week; this reduced the time to ~ 14s
i have a patch on bug 1039940 which brings this down to ~ 10s

the real gains are to be had in bug 1061233, where i'm rewriting the serialisation code from scratch.

Comment 4

[Ed Morley [:emorley]]

With bug 880097 and bug 1039940 landed, the query in comment 0 took 84-85s, vs the 76s of comment 0.
That said, the bzapi query now takes 7.0-7.5s rather than 6.5s, presumably since the number of bugs returned by both has increased since this bug was filed.

Even so, it unfortunately seems like the 50% reduction locally in comment 3 hasn't been seen in production, which is a bit odd.

Comment 5

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #4)
> With bug 880097 and bug 1039940 landed, the query in comment 0 took 84-85s,
> vs the 76s of comment 0.

s/took/now takes/

Comment 6

[Dylan Hardison [:dylan] (he/him)]

See what you've made me do, Ed! Profiling things is very addictive to me, so I've gone and profiled this.

http://people.mozilla.org/~dhardison/nytprof/bzapi/

It looks like if we add by_name => 1 to the call to Bugzilla->fields() we can save 91 seconds.

Many other parts that are slow in this are known problems and not specific to bzapi.

Comment 7

[Dylan Hardison [:dylan] (he/him)]

I've taken fix_bug from 226s to 692ms

http://people.mozilla.org/~dhardison/nytprof/bzapi-prof
http://people.mozilla.org/~dhardison/nytprof/bzapi-prof2
http://people.mozilla.org/~dhardison/nytprof/bzapi-prof3
http://people.mozilla.org/~dhardison/nytprof/bzapi-prof4
http://people.mozilla.org/~dhardison/nytprof/bzapi-prof5
http://people.mozilla.org/~dhardison/nytprof/bzapi-prof6
http://people.mozilla.org/~dhardison/nytprof/bzapi-prof7

How's them apples?

The patch is horrible and it needs to be actually cleaned up to work, but I'm tired and I didn't get anything else done today.

Comment 8

[Dylan Hardison [:dylan] (he/him)]

Attachment: hacks.patch

Comment 9

[Dylan Hardison [:dylan] (he/him)]

Attachment: hacks.patch

Sorry, I meant I changed it from 226s to 184ms

http://people.mozilla.org/~dhardison/nytprof/bzapi-prof8/

Comment 10

[Ed Morley [:emorley]]

That's awesome :-)

Comment 11

[Dylan Hardison [:dylan] (he/him)]

Attachment: 1035804_1.patch

Comment 13

[David Lawrence [:dkl]]

Comment on attachment 8791256 [details] [diff] [review]
1035804_1.patch

Review of attachment 8791256 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl

::: extensions/BzAPI/lib/Util.pm
@@ +95,3 @@
>          my @new_groups;
>          foreach my $group (@{ $data->{groups} }) {
> +            if (my $object= Bugzilla::Group->new({ name => $group })) {

if (my $object= Bugzilla::Group->new({ name => $group, cache => 1 })) {

Comment 14

[Dylan Hardison [:dylan] (he/him)]

To [e-mail]:mozilla-bteam/bmo.git
   ad29ade..3078746  master -> master

Comment 15

[Ed Morley [:emorley]]

Thank you for fixing this :-)

I'd be interested to see some webhead stats in a few days to see whether this helped at all (though I guess likely lost in the noise, since presume you don't have per transaction type stats a la New Relic).

Comment 16

[Dylan Hardison [:dylan] (he/him)]

We have per request stats but there's too much noise because different requests take such drastically different times.
There should be an overall improvement as more webheads are available to handle these faster requests.

Comment 17

[Ed Morley [:emorley]]

Hopefully :-)

(There isn't a Perl agent for NR, so it's somewhat a moot point, but this is the kind thing we have for Treeherder https://snag.gy/ALczOm.jpg)