Closed
Bug 1370484
Opened 7 years ago
Closed 7 years ago
Simplify bugzilla query in cron job
Categories
(Socorro :: Backend, task)
Socorro
Backend
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: adrian, Assigned: adrian)
References
Details
Attachments
(1 file)
In our bugzilla cron job, we query the following URL: 'https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&short_' 'desc_type=allwordssubstr&short_desc=&long_desc_type=allwordssubstr&lo' 'ng_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteb' 'oard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&ke' 'ywords=&deadlinefrom=&deadlineto=&emailassigned_to1=1&emailtype1=subs' 'tring&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1' '&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&vo' 'tes=&chfieldfrom=%s&chfieldto=Now&chfield=[Bug+creation]&chfield=reso' 'lution&chfield=bug_status&chfield=short_desc&chfield=cf_crash_signatu' 're&chfieldvalue=&cmdtype=doit&order=Importance&field0-0-0=noop&type0-' '0-0=noop&value0-0-0=&columnlist=bug_id,bug_status,resolution,short_de' 'sc,cf_crash_signature&ctype=csv' This URL is unreadable and contains a lot of unneeded fields. After running some tests, I am able to reduce it to the following: https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&chfieldfrom=%s&chfieldto=Now&chfield=[Bug+creation]&chfield=resolution&chfield=bug_status&chfield=short_desc&chfield=cf_crash_signature&columnlist=bug_id,bug_status,resolution,short_desc,cf_crash_signature&ctype=csv And after bug 1368498 lands, we won't even need to listen to changes in bug status, resolution or description, so we can simplify it even further (and reduce the number of operations we run): https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&chfieldfrom=%s&chfieldto=Now&chfield=[Bug+creation]&chfield=cf_crash_signature&columnlist=bug_id,cf_crash_signature&ctype=csv
Comment 1•7 years ago
|
||
csv?? Is it not possible to use the REST JSON endpoint instead? I remember that dkl said that any advanced search you can do you can convert to a REST JSON endpoint simply by changing the path or something.
Comment 2•7 years ago
|
||
The in_review.py script I use gets back JSON: https://github.com/willkg/socorro-zero/blob/master/bin/in_review.py The code is terrible for reasons. But maybe that's helpful? Also, there's a bugzilla python library that also gets back json: https://github.com/gdestuynder/simple_bugzilla I wouldn't switch to that library without someone doing some work on it for better viability, though (no tests, no install_requires in setup, no indication of what Python versions it supports, and so on). If that's interesting, I'm game for spending some time on it.
Assignee | ||
Comment 3•7 years ago
|
||
Comment 4•7 years ago
|
||
Commit pushed to master at https://github.com/mozilla-services/socorro https://github.com/mozilla-services/socorro/commit/62286e3fb3dda24b2bfd02e4d005125ab7635422 Fixes bug 1370484 - Refactored bugzilla query to use REST API and JSON. (#3809) * Fixes bug 1370484 - Refactored bugzilla query to use REST API and JSON. This query used to be a very long, hand-written URL, using the old bugzilla query system that returned CSV data. With this, it now uses the REST API instead, manipulating JSON data with the requests library. The query is much simpler to read and has only the few arguments it actually needs. Also, the query is not configurable anymore, because that option was never used and it would have made the code less maintainable. * Added a clear exception when bugzilla fails. * Use requests.Response.raise_for_status for HTTP errors.
Updated•7 years ago
|
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•