1034524 - Switch bzcache to the bzapi compatibility layer

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Ed Morley [:emorley]]

OF switched in bug 1026557, now time to do bzcache:
https://github.com/jonallengriffin/bzcache/blob/master/bzcache/config.py#L2

Comment 1

[Ed Morley [:emorley]]

Attachment: Switch to the native bzapi compatibility layer & remove hard-coded URL

The PR includes two commits:
1) Remove a straggler from bug 869652's conversion to using a central config.py rather than hardcoded values.
2) Switch to the native bzapi compatibility layer.

I'm happy with the deployment of this on brasstacks - however I guess we'll also need to release a new version on pypi? (for local orangefactor development etc)

Comment 2

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8450879 [details] [review]
Switch to the native bzapi compatibility layer & remove hard-coded URL

Merged as https://github.com/jonallengriffin/bzcache/commit/e684dbde9c148632a9a446f5b676927891546f1b

Comment 3

[Jonathan Griffin (:jgriffin)]

(In reply to Ed Morley [:edmorley UTC+0] from comment #1)

> I'm happy with the deployment of this on brasstacks - however I guess we'll
> also need to release a new version on pypi? (for local orangefactor
> development etc)

It looks like we've never published bzcache on PyPI before, but I just did so with a new version number:

https://pypi.python.org/pypi/bzcache/0.1.1

Comment 4

[Ed Morley [:emorley]]

[[e-mail]3 ~]$ cd ~/apps/orangefactor/src/bzcache/
[[e-mail]3 bzcache]$ git pull -v
remote: Counting objects: 12, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 12 (delta 1), reused 0 (delta 0)
Unpacking objects: 100% (12/12), done.
From https://github.com/jonallengriffin/bzcache
   672c864..553bbf5  master     -> origin/master
Updating 672c864..553bbf5
Fast-forward
 bzcache/bzload.py |    3 ++-
 bzcache/config.py |    2 +-
 setup.py          |    2 +-
 3 files changed, 4 insertions(+), 3 deletions(-)

[[e-mail]3 bzcache]$ python setup.py build
<snip>
[[e-mail]3 bzcache]$ python setup.py bdist
<snip>

[[e-mail]3 bzcache]$ crontab -l | grep bzcache
0 0,4,8,12,16,18,20 * * * /home/webtools/apps/orangefactor/bin/python /home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py

[[e-mail]3 bzcache]$ cat /home/webtools/apps/orangefactor/src/bzcache/bzcache/config.py
DEFAULT_ES_SERVER = 'elasticsearch-zlb.webapp.scl3.mozilla.com:9200'
DEFAULT_BZAPI_SERVER = 'https://bugzilla.mozilla.org/bzapi/'
DEFAULT_OF_SERVER = 'http://brasstacks.mozilla.com/orangefactor/api/'

Comment 5

[Ed Morley [:emorley]]

However, when testing this bz_cache_refresh.py seems to timeout - which if I kill early, gives:

(orangefactor)[[e-mail]3 bzcache]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh. py

♥Traceback (most recent call last):
  File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 56, in <module>
    main(options)
  File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 23, in main
    bzcache.index_bugs_by_keyword('intermittent-failure')
  File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bzcache.py", line 54, in index_bugs_by_keyword
    jsonurl = urllib.urlopen(apiURL)
  File "/usr/lib64/python2.6/urllib.py", line 86, in urlopen
    return opener.open(url)
  File "/usr/lib64/python2.6/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/usr/lib64/python2.6/urllib.py", line 444, in open_https
    errcode, errmsg, headers = h.getreply()
  File "/usr/lib64/python2.6/httplib.py", line 1064, in getreply
    response = self._conn.getresponse()
  File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse
    response.begin()
  File "/usr/lib64/python2.6/httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status
    line = self.fp.readline()
  File "/usr/lib64/python2.6/socket.py", line 433, in readline
    data = recv(1)
  File "/usr/lib64/python2.6/ssl.py", line 215, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.6/ssl.py", line 136, in read
    return self._sslobj.read(len)
KeyboardInterrupt

Or if I leave it for a while:

(orangefactor)[[e-mail]3 ~]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py
Traceback (most recent call last):
  File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 56, in <module>
    main(options)
  File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh.py", line 23, in main
    bzcache.index_bugs_by_keyword('intermittent-failure')
  File "/home/webtools/apps/orangefactor/src/bzcache/bzcache/bzcache.py", line 63, in index_bugs_by_keyword
    bug['whiteboard'],
KeyError: 'whiteboard'

Switching back to the old bzapi endpoint works:

(orangefactor)[[e-mail]3 bzcache]$ git checkout 09c7ee6687a923241fd4fd6a9590224cbef349c3
<snip>
HEAD is now at 09c7ee6... Bug 1034524 - bzload.py should use the config.py bzapi URL rather than hard-coding; r=jgriffin

(orangefactor)[[e-mail]3 bzcache]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh. py
2014-07-08, 06:05:23 - 989240 added, status: UNCONFIRMED, id: 989240
2014-07-08, 06:05:23 - 961107 added, status: NEW, id: 961107
2014-07-08, 06:05:23 - 928677 added, status: NEW, id: 928677
2014-07-08, 06:05:23 - 984274 added, status: NEW, id: 984274
...

We're now using something like:
https://bugzilla.mozilla.org/bzapi/bug?keywords=intermittent-failure&include_fields=id,summary,status,whiteboard&changed_after=2014-01-09

Compared to the previous:
https://api-dev.bugzilla.mozilla.org/latest/bug?keywords=intermittent-failure&include_fields=id,summary,status,whiteboard&changed_after=2014-01-09

It appears that the new bzapi compatibility layer:
a) Takes 76s instead of 6s to return the same results
b) Only returns the 'whiteboard' field if it is non-empty, which doesn't match bzapi behaviour.

I'll file deps for these, and for now roll back bzcache on brasstacks to 09c7ee6687a923241fd4fd6a9590224cbef349c3 so the cron can continue to work.

Comment 6

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #5)
> It appears that the new bzapi compatibility layer:
> a) Takes 76s instead of 6s to return the same results

Bug 1035804

> b) Only returns the 'whiteboard' field if it is non-empty, which doesn't
> match bzapi behaviour.

Bug 1035799

Comment 7

[Mark Côté [:mcote]]

Given that bug 1035804 is apparently going to be difficult to solve, I recommend switching that query over to the native REST API if it's not too difficult.

Comment 8

[Ed Morley [:emorley]]

If switching from bzapi to native rest, we'd need to change...

Config.py:
https://github.com/jonallengriffin/bzcache/blob/master/bzcache/config.py#L2

Then the queries that would need changing:

https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bzcache.py#L50
  def index_bugs_by_keyword(self, keyword):
    # only look at bugs that have been updated in the last 6 months
    ago = datetime.date.today() - datetime.timedelta(days=180)
    apiURL = self.bzapi_server + "bug?keywords=%s&include_fields=id,summary,status,whiteboard&changed_after=%s" % (keyword, ago.strftime('%Y-%m-%d'))
    jsonurl = urllib.urlopen(apiURL)
    buginfo = jsonurl.read()

https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bzcache.py#L66
  def _get_bugzilla_data(self, bugid_array):
    buginfo = {}
    retVal = {}
    apiURL = (self.bzapi_server + "bug?id=" + ','.join(bugid_array) +
              "&include_fields=id,summary,status,whiteboard")
    jsonurl = urllib.urlopen(apiURL)
    buginfo = jsonurl.read()

https://github.com/jonallengriffin/bzcache/blob/master/bzcache/bzload.py
  def _load_component(self, product, component, status):
    status.insert(0, '')
    now = datetime.datetime.now() - datetime.timedelta(days=180)
    url = '%sbug?product=%s&component=%s%s&include_fields=id,summary,status&changed_after=%s' % \
          (self.bzurl, product, urllib.quote(component), '&status='.join(status), now.strftime('%Y-%m-%d'))
    data = self._load_json_url(url)

Comment 9

[Mark Côté [:mcote]]

Well, we *could* mix the two to limit the number of changes, either through a second config variable, or changing the existing one to just be the URL to bmo (to which we would internally append 'bzapi' or 'rest').  Not sure that's worth it, though, since there aren't many queries to change to switch fully over to native REST.

Comment 10

[Ed Morley [:emorley]]

With bug 1035799 fixed, the Bugzilla query works, it just takes ~80s instead of ~7s. However, the actual insertion into ES still takes much longer than either of these (several minutes), so it really doesn't impact the overall runtime as much as one would expect. Seeing as the script doesn't timeout, and bug 1035804 is still looking a long ways out, I've decided to land this anyway.

[[e-mail]3 ~]$ cd ~/apps/orangefactor/src/bzcache/
[[e-mail]3 bzcache]$ source /home/webtools/apps/orangefactor/bin/activate
(orangefactor)[[e-mail]3 bzcache]$ git checkout master
Previous HEAD position was 09c7ee6... Bug 1034524 - bzload.py should use the config.py bzapi URL rather than hard-coding; r=jgriffin
Switched to branch 'master'
(orangefactor)[[e-mail]3 bzcache]$ git pull
Already up-to-date.

(orangefactor)[[e-mail]3 bzcache]$ python ~/apps/orangefactor/src/bzcache/bzcache/bz_cache_refresh. py
No handlers could be found for logger "pyes.urllib3.connectionpool"
2014-09-16, 01:49:50 - 453969 added, status: NEW, id: 453969
2014-09-16, 01:49:50 - 482073 added, status: RESOLVED, id: 482073
2014-09-16, 01:49:50 - 492821 added, status: RESOLVED, id: 492821
2014-09-16, 01:49:50 - 493692 added, status: NEW, id: 493692
2014-09-16, 01:49:50 - 497884 added, status: RESOLVED, id: 497884
2014-09-16, 01:49:50 - 501383 added, status: RESOLVED, id: 501383
<snip>
2014-09-16, 01:53:29 - 1067555 added, status: NEW, id: 1067555
2014-09-16, 01:53:29 - 1067559 added, status: NEW, id: 1067559
2014-09-16, 01:53:29 - 1067568 added, status: NEW, id: 1067568
2014-09-16, 01:53:29 - 1067569 added, status: NEW, id: 1067569
2014-09-16, 01:53:29 - 1067727 added, status: NEW, id: 1067727
2014-09-16, 01:53:29 - 1067762 added, status: NEW, id: 1067762
2014-09-16 01:53:29.445910 http://brasstacks.mozilla.com/orangefactor/api/bybug?startday=2014-07-18&endday=2014-09-16&tree=All