Closed Bug 929819 Opened 13 years ago Closed 12 years ago

[mozpool-client] TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86_64
Windows 7
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Callek, Assigned: coop)

Details

Attachments

(2 files)

[mozharness] Don't close an empty request
1.22 KB, patch
jlund
: review+
coop
: checked-in+
Details | Diff | Splinter Review
[mozpoolclient] Don't close an empty request
1.67 KB, patch
dustin
: review+
coop
: checked-in+
Details | Diff | Splinter Review
Noticed this while scanning through dead pandas this week: 17:07:29 INFO - Got request, url=http://mobile-imaging-003.p3.releng.scl1.mozilla.com/api/request/523511/ 17:07:29 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:08:29 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:09:29 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:10:29 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:11:29 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:12:29 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:13:29 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:14:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:15:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:16:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:17:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:18:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:19:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:20:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:21:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:22:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:23:30 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:24:31 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:25:31 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:26:31 INFO - Waiting for request 'ready' stage. Current state: 'failed_bad_device' 17:27:31 ERROR - INFRA-ERROR: Request did not become ready in time 17:27:31 ERROR - # TBPL EXCEPTION # 17:27:31 INFO - Aborting mozpool request. 17:27:31 INFO - Request 'http://mobile-imaging-003.p3.releng.scl1.mozilla.com/api/request/523511/' deleted on cleanup 17:27:31 FATAL - Retries limit exceeded 17:27:31 FATAL - Running post_fatal callback... 17:27:31 INFO - Running post-action listener: _resource_record_post_action 17:27:31 FATAL - Uncaught exception: Traceback (most recent call last): 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1112, in run 17:27:31 FATAL - self.run_action(action) 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1054, in run_action 17:27:31 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 995, in _possibly_run_method 17:27:31 FATAL - return getattr(self, method_name)() 17:27:31 FATAL - File "scripts/scripts/android_panda_talos.py", line 130, in request_device 17:27:31 FATAL - self.retrieve_android_device(b2gbase="") 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/mozilla/testing/mozpool.py", line 99, in retrieve_android_device 17:27:31 FATAL - self._wait_for_request_ready() 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/mozilla/testing/mozpool.py", line 132, in _wait_for_request_ready 17:27:31 FATAL - tbpl_status=TBPL_EXCEPTION, fail_cb=on_fail): 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/mozilla/testing/mozpool.py", line 122, in _retry_sleep 17:27:31 FATAL - self.fatal('Retries limit exceeded') 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 112, in fatal 17:27:31 FATAL - self.log(message, level=FATAL, exit_code=exit_code) 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 55, in log 17:27:31 FATAL - post_fatal_callback=self._post_fatal, 17:27:31 FATAL - File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 340, in log_message 17:27:31 FATAL - post_fatal_callback(message=message, exit_code=exit_code) 17:27:31 FATAL - File "scripts/scripts/android_panda_talos.py", line 388, in _post_fatal 17:27:31 FATAL - self.close_request() 17:27:31 FATAL - File "scripts/scripts/android_panda_talos.py", line 392, in close_request 17:27:31 FATAL - mph.close_request(self.request_url) 17:27:31 FATAL - File "/builds/panda-0276/test/build/venv/lib/python2.7/site-packages/mozpoolclient/mozpoolclient.py", line 248, in close_request 17:27:31 FATAL - request_url = request_url + 'return/' 17:27:31 FATAL - TypeError: unsupported operand type(s) for +: 'NoneType' and 'str' 17:27:31 FATAL - Running post_fatal callback... 17:27:31 INFO - Running post-run listener: _resource_record_post_run Traceback (most recent call last): File "scripts/scripts/android_panda_talos.py", line 406, in <module> pandaTalosTest.run_and_exit() File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1136, in run_and_exit sys.exit(self.run()) File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1114, in run self.fatal("Uncaught exception: %s" % traceback.format_exc()) File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 112, in fatal self.log(message, level=FATAL, exit_code=exit_code) File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 55, in log post_fatal_callback=self._post_fatal, File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 340, in log_message post_fatal_callback(message=message, exit_code=exit_code) File "scripts/scripts/android_panda_talos.py", line 388, in _post_fatal self.close_request() File "scripts/scripts/android_panda_talos.py", line 392, in close_request mph.close_request(self.request_url) File "/builds/panda-0276/test/build/venv/lib/python2.7/site-packages/mozpoolclient/mozpoolclient.py", line 248, in close_request request_url = request_url + 'return/' TypeError: unsupported operand type(s) for +: 'NoneType' and 'str' -- We likely should also break out of the retries when we get failed_bad_device
(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #1) > Should we just not call close_request if self.request_url is None? > http://hg.mozilla.org/build/mozharness/file/7bf07524c01b/scripts/ > android_panda_talos.py#l392 Sounds like a simple, sane fix. Any takers?
Assignee: nobody → coop
Status: NEW → ASSIGNED
Priority: -- → P2
Attachment #8470214 - Flags: review?(dustin) → review+
Comment on attachment 8470213 [details] [diff] [review] [mozharness] Don't close an empty request Review of attachment 8470213 [details] [diff] [review]: ----------------------------------------------------------------- lgtm, I'm guessing we should give the same love to the unittest equivalent: http://mxr.mozilla.org/build/source/mozharness/scripts/android_panda.py#472 ?
Attachment #8470213 - Flags: review?(jlund) → review+
Comment on attachment 8470213 [details] [diff] [review] [mozharness] Don't close an empty request Review of attachment 8470213 [details] [diff] [review]: ----------------------------------------------------------------- (In reply to Jordan Lund (:jlund) from comment #5) > Comment on attachment 8470213 [details] [diff] [review] > [mozharness] Don't close an empty request > > Review of attachment 8470213 [details] [diff] [review]: > ----------------------------------------------------------------- > > lgtm, I'm guessing we should give the same love to the unittest equivalent: > http://mxr.mozilla.org/build/source/mozharness/scripts/android_panda.py#472 ? Landed with that additional change: https://hg.mozilla.org/build/mozharness/rev/e29a6d6e3450
Attachment #8470213 - Flags: checked-in+
Comment on attachment 8470214 [details] [diff] [review] [mozpoolclient] Don't close an empty request Review of attachment 8470214 [details] [diff] [review]: ----------------------------------------------------------------- https://hg.mozilla.org/build/mozpool/rev/f2d647b2c274
Attachment #8470214 - Flags: checked-in+
Merged to production, and deployed [mozharness].
Still need to follow the instructions to deploy a new version of the client: https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client
(In reply to Chris Cooper [:coop] from comment #9) > Still need to follow the instructions to deploy a new version of the client: > > https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client Callek: Per these instructions, does http://hg.mozilla.org/build/tools/lib/python/vendor/mozpoolclient-0.1.4 need updating?
Flags: needinfo?(bugspam.Callek)
Dustin: the new package is uploaded to pypi[1], but the package maintainer instructions aren't linked to anything in https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client. Are there next steps here to start using the new package? 1. http://pypi.pvt.build.mozilla.org/pub/mozpoolclient-0.1.6.tar.gz
Flags: needinfo?(dustin)
(In reply to Chris Cooper [:coop] from comment #10) > (In reply to Chris Cooper [:coop] from comment #9) > > Still need to follow the instructions to deploy a new version of the client: > > > > https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client > > Callek: Per these instructions, does > http://hg.mozilla.org/build/tools/lib/python/vendor/mozpoolclient-0.1.4 need > updating? Not sure who added me as a gating factor for that code, imo, its always worth the change here, unless its a breaking API change in which case we might have to change a few consumers in tools we should update it (after all, if it is a breaking change we can rollback with an hg backout anyway)
Flags: needinfo?(bugspam.Callek)
(In reply to Justin Wood (:Callek) from comment #12) > Not sure who added me as a gating factor for that code, imo, its always > worth the change here, unless its a breaking API change in which case we > might have to change a few consumers in tools we should update it (after > all, if it is a breaking change we can rollback with an hg backout anyway) The copy of mozpoolclient in the tools repo has been updated to 0.1.6: https://hg.mozilla.org/build/tools/rev/056d0e0021aa
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Flags: needinfo?(dustin)
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: