964620 - Increase mozpoolclient timeout for creating a new request

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

Okay, I don't actually know the percentage, because really counting is annoying. On the bad end, https://tbpl.mozilla.org/?rev=4da3e21a0e5f&jobname=panda set RETRY on 18 of 33 first attempts, on the not so bad end https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=4a1e002cb3c2&jobname=panda set RETRY on 4 first attempts.

They all seem to be of the form of https://tbpl.mozilla.org/php/getParsedLog.php?id=33666407&tree=Mozilla-Central

19:38:29    ERROR - Try 1: Can't get http://mobile-imaging-002.p2.releng.scl1.mozilla.com/api/device/panda-0233/request/: HTTPConnectionPool(host='mobile-imaging-002.p2.releng.scl1.mozilla.com', port=80): Request timed out. (timeout=10)!
19:38:29    ERROR - Traceback (most recent call last):
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/mozpoolclient.py", line 128, in url_post
19:38:29    ERROR -     r = requests.post(url, data=data, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/api.py", line 88, in post
19:38:29    ERROR -     return request('post', url, data=data, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/api.py", line 44, in request
19:38:29    ERROR -     return session.request(method=method, url=url, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
19:38:29    ERROR -     resp = self.send(prep, **send_kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
19:38:29    ERROR -     r = adapter.send(request, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/adapters.py", line 333, in send
19:38:29    ERROR -     raise Timeout(e)
19:38:29    ERROR - 
19:38:29    ERROR - Timeout: HTTPConnectionPool(host='mobile-imaging-002.p2.releng.scl1.mozilla.com', port=80): Request timed out. (timeout=10)
19:38:29    ERROR - 
19:38:29     INFO - Sleeping 2...
19:38:38     INFO - Got request, url=http://mobile-imaging-002.p2.releng.scl1.mozilla.com/api/request/839073/
19:38:39     INFO - Waiting for request 'ready' stage.  Current state: 'finding_device'
19:39:39     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:40:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:41:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:42:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:43:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:44:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:45:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:46:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:47:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:48:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:49:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:50:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:51:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:52:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:53:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:54:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:55:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:56:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:57:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:58:42    ERROR - INFRA-ERROR: Request did not become ready in time
19:58:42    ERROR - # TBPL EXCEPTION #
19:58:42     INFO - Aborting mozpool request.
19:58:42     INFO - Request 'http://mobile-imaging-002.p2.releng.scl1.mozilla.com/api/request/839073/' deleted on cleanup
19:58:42    FATAL - Retries limit exceeded
19:58:42    FATAL - Running post_fatal callback...
19:58:42    FATAL - Exiting -1

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

This is due to bug 948244.  I can't find the parent bug to that, but there was a plan to increase the timeout in mozpoolclient to >10s.

I'll ship a new version of Mozpool today.  Let's leave this open to increase the timeout.

Comment 2

[Armen [:armenzg]]

Attachment: bump mozpoolclient's timeout

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8366628 [details] [diff] [review]
bump mozpoolclient's timeout

Ah, I thought this was a mozharness script somewhere.

Comment 4

[Armen [:armenzg]]

Attachment: increase timeout + bump the version

I forgot to bump the version and take note of the changes

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8367436 [details] [diff] [review]
increase timeout + bump the version

good catch

Comment 6

[Armen [:armenzg]]

Attachment: update mozpoolclient for the tools repo

I also added CHANGES.txt and __init__.py

I added the steps for this in here:
https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client

Comment 7

[Armen [:armenzg]]

Comment on attachment 8367436 [details] [diff] [review]
increase timeout + bump the version

http://hg.mozilla.org/build/mozpool/rev/359aa1c289ae

[[e-mail]3 armenzg]# ls -l /mnt/netapp/relengweb/pypi/pub/mozpoolclient-0.1.5.tar.gz 
-rw-r--r-- 1 armenzg armenzg 5122 Jan 29 11:07 /mnt/netapp/relengweb/pypi/pub/mozpoolclient-0.1.5.tar.gz

armenzg-thinkpad vendor hg:[default!] $ curl -I http://pypi.pub.build.mozilla.org/pub/mozpoolclient-0.1.5.tar.gz
HTTP/1.1 200 OK
Date: Wed, 29 Jan 2014 19:51:37 GMT
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Last-Modified: Wed, 29 Jan 2014 19:07:59 GMT
ETag: "1402"
Accept-Ranges: bytes
Content-Length: 5122
Content-Type: application/x-gzip

Comment 8

[Armen [:armenzg]]

Attachment: update mozpoolclient for the tools repo + bump version

And bumping http://mxr.mozilla.org/build/source/tools/lib/python/vendorlibs.pth

Comment 9

[Armen [:armenzg]]

Comment on attachment 8367516 [details] [diff] [review]
update mozpoolclient for the tools repo + bump version

https://hg.mozilla.org/build/tools/rev/e259cd859290