1173855 - Autophone - check for timeout in PerfTest.publish_results should test for timeout message

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

Example email shows the timeout does not retry but fails after the first timeout.

There was an error attempting to send test results to the result server http://phonedash.mozilla.org.

Job        https://treeherder.mozilla.org/#/jobs?filter-searchStr=autophone&exclusion_profile=false&repo=mozilla-inbound&revision=321a1148b604
Test       autophone-s1s2
Phone      nexus-s-2
Repository mozilla-inbound
Build      20150610221245
Revision   http://hg.mozilla.org/integration/mozilla-inbound/rev/321a1148b604
Exception  <urlopen error [Errno 60] Operation timed out>
Result     {
  "blddate": 1433999565, 
  "bldtype": "opt", 
  "cached": false, 
  "machineid": "Nexus S", 
  "osver": "2.3.6", 
  "phoneid": "nexus-s-2", 
  "productname": "org.mozilla.fennec", 
  "productversion": "41.0a1", 
  "rejected": false, 
  "revision": "http://hg.mozilla.org/integration/mozilla-inbound/rev/321a1148b604", 
  "starttime": 148.0, 
  "testname": "local-blank", 
  "throbberstart": 1355.0, 
  "throbberstop": 5418.0
}

https://github.com/bclary/autophone/blob/master/tests/perftest.py#L188 uses a check for errno == 60 to determine if the urlopen failed due to a timeout. This is incorrect as the urllib2.URLError doesn't have a meaningful errno. A better check is to test if a string is found in the exception, e.g. 'timed out' in e

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: bug-1173855-urlopen-timeout.patch

>>> import urllib2
>>> e = urllib2.URLError('timed out')
>>> 'timed out' in e
True
>>> 'yoda' in e
False

Comment 2

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/c9e7eaa8afbbfe69b8e565019a584219c8249598

Comment 3

[Bob Clary [:bc] (inactive)]

This didn't work.

Comment 4

[Bob Clary [:bc] (inactive)]

Attachment: bug-1173855-followup.patch

Just treat all exceptions the same. I've been testing this in production this morning and it appears to work fine.

Comment 5

[Geoff Brown [:gbrown]]

Comment on attachment 8621727 [details] [diff] [review]
bug-1173855-followup.patch

Review of attachment 8621727 [details] [diff] [review]:
-----------------------------------------------------------------

That's certainly simpler and less hacky.

Do you know why the "timed out" check was not working?

Comment 6

[Bob Clary [:bc] (inactive)]

Nope. :-(

Comment 7

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/90a630500695b4aedd55f1b0403630793fc792bb