Closed Bug 964620 Opened 8 years ago Closed 8 years ago

Increase mozpoolclient timeout for creating a new request

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

ARM
Android
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: philor, Assigned: armenzg)

Details

Attachments

(2 files, 2 obsolete files)

Okay, I don't actually know the percentage, because really counting is annoying. On the bad end, https://tbpl.mozilla.org/?rev=4da3e21a0e5f&jobname=panda set RETRY on 18 of 33 first attempts, on the not so bad end https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=4a1e002cb3c2&jobname=panda set RETRY on 4 first attempts.

They all seem to be of the form of https://tbpl.mozilla.org/php/getParsedLog.php?id=33666407&tree=Mozilla-Central

19:38:29    ERROR - Try 1: Can't get http://mobile-imaging-002.p2.releng.scl1.mozilla.com/api/device/panda-0233/request/: HTTPConnectionPool(host='mobile-imaging-002.p2.releng.scl1.mozilla.com', port=80): Request timed out. (timeout=10)!
19:38:29    ERROR - Traceback (most recent call last):
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/mozpoolclient.py", line 128, in url_post
19:38:29    ERROR -     r = requests.post(url, data=data, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/api.py", line 88, in post
19:38:29    ERROR -     return request('post', url, data=data, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/api.py", line 44, in request
19:38:29    ERROR -     return session.request(method=method, url=url, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
19:38:29    ERROR -     resp = self.send(prep, **send_kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
19:38:29    ERROR -     r = adapter.send(request, **kwargs)
19:38:29    ERROR - 
19:38:29    ERROR -   File "/builds/panda-0233/test/build/venv/lib/python2.7/site-packages/requests/adapters.py", line 333, in send
19:38:29    ERROR -     raise Timeout(e)
19:38:29    ERROR - 
19:38:29    ERROR - Timeout: HTTPConnectionPool(host='mobile-imaging-002.p2.releng.scl1.mozilla.com', port=80): Request timed out. (timeout=10)
19:38:29    ERROR - 
19:38:29     INFO - Sleeping 2...
19:38:38     INFO - Got request, url=http://mobile-imaging-002.p2.releng.scl1.mozilla.com/api/request/839073/
19:38:39     INFO - Waiting for request 'ready' stage.  Current state: 'finding_device'
19:39:39     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:40:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:41:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:42:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:43:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:44:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:45:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:46:40     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:47:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:48:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:49:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:50:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:51:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:52:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:53:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:54:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:55:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:56:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:57:41     INFO - Waiting for request 'ready' stage.  Current state: 'failed_device_busy'
19:58:42    ERROR - INFRA-ERROR: Request did not become ready in time
19:58:42    ERROR - # TBPL EXCEPTION #
19:58:42     INFO - Aborting mozpool request.
19:58:42     INFO - Request 'http://mobile-imaging-002.p2.releng.scl1.mozilla.com/api/request/839073/' deleted on cleanup
19:58:42    FATAL - Retries limit exceeded
19:58:42    FATAL - Running post_fatal callback...
19:58:42    FATAL - Exiting -1
This is due to bug 948244.  I can't find the parent bug to that, but there was a plan to increase the timeout in mozpoolclient to >10s.

I'll ship a new version of Mozpool today.  Let's leave this open to increase the timeout.
Summary: Panda jobs are 25-50% RETRY from Can't get http://mobile-imaging-009.p9.releng.scl1.mozilla.com/api/device/panda-0850/request/: HTTPConnectionPool(host='mobile-imaging-009.p9.releng.scl1.mozilla.com', port=80): Request timed out → Increase mozpoolclient timeout for creating a new request
Attached patch bump mozpoolclient's timeout (obsolete) — Splinter Review
Assignee: nobody → armenzg
Status: NEW → ASSIGNED
Attachment #8366628 - Flags: review?(dustin)
Comment on attachment 8366628 [details] [diff] [review]
bump mozpoolclient's timeout

Ah, I thought this was a mozharness script somewhere.
Attachment #8366628 - Flags: review?(dustin) → review+
I forgot to bump the version and take note of the changes
Attachment #8366628 - Attachment is obsolete: true
Attachment #8367436 - Flags: review?(dustin)
Comment on attachment 8367436 [details] [diff] [review]
increase timeout + bump the version

good catch
Attachment #8367436 - Flags: review?(dustin) → review+
I also added CHANGES.txt and __init__.py

I added the steps for this in here:
https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client
Attachment #8367514 - Flags: review?(bugspam.Callek)
Comment on attachment 8367436 [details] [diff] [review]
increase timeout + bump the version

http://hg.mozilla.org/build/mozpool/rev/359aa1c289ae

[root@relengwebadm.private.scl3 armenzg]# ls -l /mnt/netapp/relengweb/pypi/pub/mozpoolclient-0.1.5.tar.gz 
-rw-r--r-- 1 armenzg armenzg 5122 Jan 29 11:07 /mnt/netapp/relengweb/pypi/pub/mozpoolclient-0.1.5.tar.gz

armenzg-thinkpad vendor hg:[default!] $ curl -I http://pypi.pub.build.mozilla.org/pub/mozpoolclient-0.1.5.tar.gz
HTTP/1.1 200 OK
Date: Wed, 29 Jan 2014 19:51:37 GMT
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Last-Modified: Wed, 29 Jan 2014 19:07:59 GMT
ETag: "1402"
Accept-Ranges: bytes
Content-Length: 5122
Content-Type: application/x-gzip
Attachment #8367436 - Flags: checked-in+
And bumping http://mxr.mozilla.org/build/source/tools/lib/python/vendorlibs.pth
Attachment #8367514 - Attachment is obsolete: true
Attachment #8367514 - Flags: review?(bugspam.Callek)
Attachment #8367516 - Flags: review?(bugspam.Callek)
Attachment #8367516 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8367516 [details] [diff] [review]
update mozpoolclient for the tools repo + bump version

https://hg.mozilla.org/build/tools/rev/e259cd859290
Attachment #8367516 - Flags: checked-in+
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.