713268 - Search calls using changed_after are not returning expected results when using the minutes option

Status: RESOLVED WONTFIX

(Webtools Graveyard :: BzAPI, defect)

Description

[Daniel Buchner [:dbuc]]

If you run the two API search queries below, you would expect the one using changed_after=1m to show less (or 0) bugs in comparison to the same query using changed_after=1h. I believe the minute resolution option for this parameter is broken, please advise.

https://api-dev.bugzilla.mozilla.org/latest/bug?changed_after=1m&component=Add-on%20Builder&product=addons.mozilla.org

https://api-dev.bugzilla.mozilla.org/latest/bug?changed_after=1h&component=Add-on%20Builder&product=addons.mozilla.org

Comment 1

[Daniel Buchner [:dbuc]]

Is there some workaround for this in the meantime, my poor Bz app is up a certain creek without a paddle until I can get minute resolution :(

Comment 2

[Gervase Markham [:gerv]]

Do you get correct results out of Bugzilla when doing an equivalent query using the web interface?

Also, I suspect the Bugzilla admins might prefer it that you didn't run queries every minute... how many are you planning, and how complex are they?

Gerv

Comment 3

[Daniel Buchner [:dbuc]]

(In reply to Gervase Markham [:gerv] from comment #2)
> Do you get correct results out of Bugzilla when doing an equivalent query
> using the web interface?

It appears minute resolution changed_after calls also return wildly incorrect results: https://bugzilla.mozilla.org/buglist.cgi?changed_after=1m&component=Add-on%20Builder&product=addons.mozilla.org

> Also, I suspect the Bugzilla admins might prefer it that you didn't run
> queries every minute... how many are you planning, and how complex are they?

I need a query at no less than 5min resolution that simply tells me if any bugs that match the params passed have changed. This can be a boolean return with no actual data. The app I have tracks incoming new bugs and changes, I need a simple heads or tails diff to do so. Is there anyway I can get this?

Comment 4

[Daniel Buchner [:dbuc]]

It should be noted that a minimum floor for this call is totally fine (I hope it is no more than 5 minutes) - but even using "30m" (or any other increment for that matter) is also borked, so this is still an issue regardless.

Comment 5

[Gervase Markham [:gerv]]

Are you sure "m" means "minute" rather than "month"?

Anyway, if Bugzilla isn't doing it right, BzAPI can't do it right - you need to file a bug on Bugzilla itself. :-) Resolving INCOMPLETE for now.

Gerv

Comment 6

[Gervase Markham [:gerv]]

Yes, "m" means month:
http://bzr.mozilla.org/bugzilla/4.2/annotate/head:/Bugzilla/Search.pm#L1945

There is no unit smaller than "h" for hour. If you want a minutes option, you'll need to file an Bugzilla bug :-)

Gerv

Comment 7

[Daniel Buchner [:dbuc]]

I will file the bug again on Bugzilla, but I wonder how Bugzilla sends out emails when things change? It suggests the logic is surely there to provide such a call. My app was looking so good too, sad face.

Comment 8

[Gervase Markham [:gerv]]

Bugzilla sends out emails when things change in an event-driven manner, not by doing regular queries. Some people track changes by becoming a "global watcher", and getting a copy of all emails Bugzilla sends. Be careful, though, that's quite a firehose. Alternatively, you could listen on clegnitto's Pulse <http://pulse.mozilla.org/> - which I believe gets the emails and provides notifications (and will soon get the data via a more reliable method than email, if it's not already).

If you tell us your problem, rather than what's broken with the solution you've chosen, we can normally help better :-))

Gerv

Comment 9

[Daniel Buchner [:dbuc]]

I have an app that tracks a user-entered set of bug queries. It needs to know when there is a change to a bug that matches any of these user-entered queries occurs for two reasons: System notification of change and the subsequent fetch of the modified bug list. A constraint, is that there is no server, this is a 100% client app that uses only HTML/CSS/JS and various web APIs to accomplish its goals. Not having the ability to receive feeds of bug changes within a query essentially eliminates all client app use-cases that need awareness of those changes in a timely manner.

I have a bug in on the Bugzilla product, and I really hope it is addressed. For us to say "Well you just need to run X server, or Y mail server to do this" is odd. This case is not an outlier, any dashboard worth its weight in feathers would need this. Besides that, we're a primarily a client-side web technology company, for us to impose such server-side constraints is IMHO a bit odd given there is already an API call that filters data in this way at a lower time resolution.

Comment 10

[Gervase Markham [:gerv]]

If you are looking for info in a timely manner, then a polling system is not the right way to go. If lots of people used that, it would bring Bugzilla down. You want a system where you are notified of specific events - which is what Pulse does. There appears to be an implementation of AMQP in JS: https://github.com/dansimpson/amqp-js so perhaps there is a client-side-only way of using the Pulse service. 

Gerv

Comment 11

[Daniel Buchner [:dbuc]]

I reviewed that project on Github, and it made no mention of Bugzilla, talked about reliance on a server adhering to its specific requirements, and uses flash, was there a section I missed that invalidates all of this? I don't understand why setting up ES with a 5 minute floor on calls and a caching layer to eliminate needless server processing and data computation would allow people crush the server, I doubt that would be the case.

Comment 12

[Gervase Markham [:gerv]]

"A server adhering to its specific requirements" is presumably an AMQP server, like pulse.mozilla.org.

I suggest you talk to clegnitto - he is best placed to be able to tell you whether or not this system meets your requirements. It's certainly what it's designed for, although the client-only requirement may make it more difficult.

I am not saying that there are no other ways to do this, I am attempting to think of ways which do not require the Bugzilla team to make changes - not because we are unwilling, but because it will enable you to make progress faster.

What is "ES"?

Gerv

Comment 13

[Daniel Buchner [:dbuc]]

Yeah I think my desire for a client-only app is probably at the crux here. I mentioned ES (Elastic Search, based on Lucene) because we use it for Add-on Builder and Metrics uses it for huge real-time data sets.

ES can index and cache results of query results and has internal push notices that invalidate caches (for instance, if a bug was updated and you wanted to invalidate the cache on a user-defined query list). The system could handle these types of calls if it essentially registered and retained a hash list of "active" bug queries, and only ever did invalidation checks on those queries once every period the user defines via changed_after. This means you can have one query come in that is set to 5 min resolution, and Bugzilla would only ever hit its DB to check for bug changes for that query 12 times in an hour, even if 12 million people were pinging for it.

Comment 14

[Daniel Buchner [:dbuc]]

"ES can index and cache results of query results" <-- wow, terrible unintended grammar error :)

Comment 15

[Gervase Markham [:gerv]]

Comment 13 sounds like a large quantity of work to fit it to Bugzilla, but if you want to push that, file a bug in the bugzilla.mozilla.org product and dkl and glob can look at it for you :-)

Gerv