1193651 - The weekly top oranges script is occasionally timing out

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Ed Morley [:emorley]]

eg:

Running in test mode.
Getting top bugs...
Getting bug data from /bybug?tree=trunk&startday=2015-07-31&endday=2015-08-06.
Getting JSON from http://localhost/orangefactor/api/bybug?tree=trunk&startday=2015-07-31&endday=2015-08-06.
Results: <snip>
...skipping...
  File "woo_mailer.py", line 70, in top_bugs
    jdata = self.get_bybug()
  File "woo_mailer.py", line 100, in get_bybug
    return self.get_bugdata('/bybug?tree=trunk&startday=%s&endday=%s' % (self.startday, self.endday))
  File "woo_mailer.py", line 95, in get_bugdata
    jdata = self.get_json(url_path)
  File "woo_mailer.py", line 86, in get_json
    data = urllib2.urlopen(url, timeout=30).read()
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error timed out>
Running in production mode.
Getting top bugs...
Getting bug data from /bybug?tree=trunk&startday=2015-08-04&endday=2015-08-10.
Getting JSON from http://localhost/orangefactor/api/bybug?tree=trunk&startday=2015-08-04&endday=2015-08-10.
Traceback (most recent call last):
  File "woo_mailer.py", line 282, in <module>
    main()
  File "woo_mailer.py", line 195, in main
    top = tb.top_bugs()[:10]
  File "woo_mailer.py", line 70, in top_bugs
    jdata = self.get_bybug()
  File "woo_mailer.py", line 100, in get_bybug
    return self.get_bugdata('/bybug?tree=trunk&startday=%s&endday=%s' % (self.startday, self.endday))
  File "woo_mailer.py", line 95, in get_bugdata
    jdata = self.get_json(url_path)
  File "woo_mailer.py", line 86, in get_json
    data = urllib2.urlopen(url, timeout=30).read()
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error timed out>

We should:
1) Actually check the wrapper shell scripts into the repo
2) Disable test mode for the prod script
3) Stop dumping so much spam to the log
4) Increase the urllib2 timeout from 30s to something larger

Comment 1

[Ed Morley [:emorley]]

Attachment: Make the weekly mailer script less susecptible to timeouts

* Stops running the mailer script twice (once in test mode, once in normal mode)
* Raises the get json timeout from 30s to 120s
* Switches to using requests over urllib2
* Retries up to 3 times if there was an error
* Now prints the get json response body only if there was an error
* Now prints the URL rather than entire cache contents when there is a cache hit
* Sorts the imports in woo_mailer.py to follow PEP8 recommendations

I'll be using parts of this mailer script for bug 1179310, so extra resilience here will improve the reliability of that script too.

Comment 2

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8646861 [details] [diff] [review]
Make the weekly mailer script less susecptible to timeouts

Review of attachment 8646861 [details] [diff] [review]:
-----------------------------------------------------------------

thanks!

Comment 3

[Ed Morley [:emorley]]

Thank you for the review :-)

[[e-mail]3 root]$ cd ~/apps/orangefactor/src/orangefactor/
[[e-mail]3 orangefactor]$ hg pull -u -v
pulling from http://hg.mozilla.org/automation/orangefactor/
searching for changes
all local heads known remotely
adding changesets
adding manifests
adding file changes
added 1 changesets with 3 changes to 3 files
resolving manifests
getting requirements/prod.txt
getting woo_mailer.py
getting woo_mailer.sh
3 files updated, 0 files merged, 0 files removed, 0 files unresolved

[[e-mail]3 orangefactor]$ ~/apps/orangefactor/bin/pip install -r requirements/prod.txt
...
Successfully installed requests-2.7.0

[[e-mail]3 orangefactor]$ crontab -l
...
0 8 * * tue     (/home/webtools/apps/orangefactor/src/orangefactor/woo_mailer.sh 2>&1) >> /home/webtools/apps/orangefactor/woo_mailer.log

[[e-mail]3 orangefactor]$ (/home/webtools/apps/orangefactor/src/orangefactor/woo_mailer.sh 2>&1) >>  /home/webtools/apps/orangefactor/woo_mailer.log

[[e-mail]3 orangefactor]$ tail ~/apps/orangefactor/woo_mailer.log -n 3
Getting JSON from http://localhost/orangefactor/api/bybug?tree=trunk&startday=2015-07-30&endday=2015-08-05.
Sending email...
Done!