1301541 - BzAPI bug search query takes twice as long as one using the native REST API

Status: RESOLVED DUPLICATE of bug 1035804

(bugzilla.mozilla.org :: Extensions, defect)

Description

[Ed Morley [:emorley]]

This may end up being wontfix, but if so, I think we need to more aggressively switch people off of the BzAPI compatibility shim, since I'm presuming it's not just slowing down consumers, but causing greater load on BMO as well.

$ PARAMS='keywords=intermittent-failure&include_fields=id,summary,status,whiteboard&chfield from=-6m&limit=500'

$ time curl -sSfo /dev/null "https://bugzilla.mozilla.org/bzapi/bug?${PARAMS}"
real    0m15.203s
user    0m0.046s
sys     0m0.046s

$ time curl -sSfo /dev/null "https://bugzilla.mozilla.org/rest/bug?${PARAMS}"
real    0m8.728s
user    0m0.046s
sys     0m0.000s

The query in bug 1301144 is using BzAPI, so this bug is one of the reasons it's so slow. (OrangeFactor uses bzcache, which has now been been partly converted over to the new REST API in  https://github.com/jonallengriffin/bzcache/pull/4)

Comment 1

[Ed Morley [:emorley]]

The increase compared to the native REST API is even more pronounced with queries that return a large number of bugs:

PARAMS='keywords=intermittent-failure&include_fields=id&chfieldfrom=-6m'

$ time curl -sSfo /dev/null "https://bugzilla.mozilla.org/bzapi/bug?${PARAMS}"
real    2m11.149s
user    0m0.031s
sys     0m0.015s

$ time curl -sSfo /dev/null "https://bugzilla.mozilla.org/rest/bug?${PARAMS}"
real    0m19.533s
user    0m0.031s
sys     0m0.031s

(both still return the same number of bugs)

Comment 2

[Dylan Hardison [:dylan] (he/him)]

It is not a good use of developer time to do this, however I'll see about splitting off the requests to a dedicated web-head.

Comment 3

[Ed Morley [:emorley]]

(In reply to Dylan Hardison [:dylan] from comment #2)
> It is not a good use of developer time to do this

I agree, however this leaves:

(In reply to Ed Morley [:emorley] from comment #0)
> This may end up being wontfix, but if so, I think we need to more
> aggressively switch people off of the BzAPI compatibility shim, since I'm
> presuming it's not just slowing down consumers, but causing greater load on
> BMO as well.

Would you prefer to track this via:
* reopening this bug
* filing a new bug
* newsgroup discussion?

Comment 4

[Ed Morley [:emorley]]

ie: 

As the unfortunate 'last person to touch' a few ancient tools that use Bugzilla's API, I was vaguely aware they used BzAPI not the REST API, but (a) wasn't 100% sure which were migrated, (b) had no idea they were experiencing a 2-6x slowdown by doing so. As such, I hadn't given it a second thought. If I had known, I would have prioritised migration to the native REST API.

Suggested actions:
* Creating (or improving an existing) "API best practices" page, which warns against using BzAPI (plus suggests general tips like using pagination, last_modified, sensible include_fields, setting a helpful User Agent etc)
* Adding a warning banner to all BzAPI resources saying EOL plus in many cases many times slower
* Blogging/emailing newsgroups apart BzAPI perf issues - pointing to new best practices page
* Performing inspection of BzAPI web logs (via referrer, query strings, User Agent) to identify top N consumers, and file bugs against them
* (Optionally, in the future, depending on usage dropping) Setting an EOL date for BzAPI entirely.

I'm happy to spend time on the web logs part fwiw.

Comment 5

[Mark Côté [:mcote]]

FYI there is a deprecation notice at the top of https://wiki.mozilla.org/Bugzilla:BzAPI, but you are right that it doesn't mention speed.  I'll update it.

Comment 6

[Dylan Hardison [:dylan] (he/him)]

We can see which users are using bzapi (For the most part) and start contacting them. I'm also going to work on forcing a limit,
and we're getting a node dedicated to bzapi queries. User education in the form of best practices is good.

It may also be obviated by the elasticsearch code. That code (which wraps the internal search API) is so much faster than bzapi that most searches should be IO-bound... A discussion on tools-bmo is probably the way of handling this.

Comment 7

[Dylan Hardison [:dylan] (he/him)]

sorry, "is much faster than the native searches", not "faster than bzapi". bzapi will make use if when it lands, without us doing anything. It will be an even better selling point, as a search for intermittent-failure on native rest using elastic search takes a tenth of a second. :-)

Comment 8

[Ed Morley [:emorley]]

So this is pretty funny - I'd forgotten about filing pretty much the same bug two years ago: bug 1035804.

Also, let's move discussion of BzAPI->native REST persuasion to bug 1003236.

Comment 9

[Dylan Hardison [:dylan] (he/him)]

performing that query results in 104,961,987 function calls. that is largest reason it is slow. I'm posting the profiling data just for funs.

Comment 10

[Ed Morley [:emorley]]

This is being fixed now in bug 1035804.