The default bug view has changed. See this FAQ.

[mozpool-client] TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

RESOLVED FIXED

Status

Release Engineering
Platform Support
P2
normal
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: Callek, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

4 years ago
Noticed this while scanning through dead pandas this week:

17:07:29     INFO - Got request, url=http://mobile-imaging-003.p3.releng.scl1.mozilla.com/api/request/523511/
17:07:29     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:08:29     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:09:29     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:10:29     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:11:29     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:12:29     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:13:29     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:14:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:15:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:16:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:17:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:18:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:19:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:20:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:21:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:22:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:23:30     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:24:31     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:25:31     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:26:31     INFO - Waiting for request 'ready' stage.  Current state: 'failed_bad_device'
17:27:31    ERROR - INFRA-ERROR: Request did not become ready in time
17:27:31    ERROR - # TBPL EXCEPTION #
17:27:31     INFO - Aborting mozpool request.
17:27:31     INFO - Request 'http://mobile-imaging-003.p3.releng.scl1.mozilla.com/api/request/523511/' deleted on cleanup
17:27:31    FATAL - Retries limit exceeded
17:27:31    FATAL - Running post_fatal callback...
17:27:31     INFO - Running post-action listener: _resource_record_post_action
17:27:31    FATAL - Uncaught exception: Traceback (most recent call last):
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1112, in run
17:27:31    FATAL -     self.run_action(action)
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1054, in run_action
17:27:31    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 995, in _possibly_run_method
17:27:31    FATAL -     return getattr(self, method_name)()
17:27:31    FATAL -   File "scripts/scripts/android_panda_talos.py", line 130, in request_device
17:27:31    FATAL -     self.retrieve_android_device(b2gbase="")
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/mozilla/testing/mozpool.py", line 99, in retrieve_android_device
17:27:31    FATAL -     self._wait_for_request_ready()
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/mozilla/testing/mozpool.py", line 132, in _wait_for_request_ready
17:27:31    FATAL -     tbpl_status=TBPL_EXCEPTION, fail_cb=on_fail):
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/mozilla/testing/mozpool.py", line 122, in _retry_sleep
17:27:31    FATAL -     self.fatal('Retries limit exceeded')
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 112, in fatal
17:27:31    FATAL -     self.log(message, level=FATAL, exit_code=exit_code)
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 55, in log
17:27:31    FATAL -     post_fatal_callback=self._post_fatal,
17:27:31    FATAL -   File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 340, in log_message
17:27:31    FATAL -     post_fatal_callback(message=message, exit_code=exit_code)
17:27:31    FATAL -   File "scripts/scripts/android_panda_talos.py", line 388, in _post_fatal
17:27:31    FATAL -     self.close_request()
17:27:31    FATAL -   File "scripts/scripts/android_panda_talos.py", line 392, in close_request
17:27:31    FATAL -     mph.close_request(self.request_url)
17:27:31    FATAL -   File "/builds/panda-0276/test/build/venv/lib/python2.7/site-packages/mozpoolclient/mozpoolclient.py", line 248, in close_request
17:27:31    FATAL -     request_url = request_url + 'return/'
17:27:31    FATAL - TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
17:27:31    FATAL - Running post_fatal callback...
17:27:31     INFO - Running post-run listener: _resource_record_post_run
Traceback (most recent call last):
  File "scripts/scripts/android_panda_talos.py", line 406, in <module>
    pandaTalosTest.run_and_exit()
  File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1136, in run_and_exit
    sys.exit(self.run())
  File "/builds/panda-0276/test/scripts/mozharness/base/script.py", line 1114, in run
    self.fatal("Uncaught exception: %s" % traceback.format_exc())
  File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 112, in fatal
    self.log(message, level=FATAL, exit_code=exit_code)
  File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 55, in log
    post_fatal_callback=self._post_fatal,
  File "/builds/panda-0276/test/scripts/mozharness/base/log.py", line 340, in log_message
    post_fatal_callback(message=message, exit_code=exit_code)
  File "scripts/scripts/android_panda_talos.py", line 388, in _post_fatal
    self.close_request()
  File "scripts/scripts/android_panda_talos.py", line 392, in close_request
    mph.close_request(self.request_url)
  File "/builds/panda-0276/test/build/venv/lib/python2.7/site-packages/mozpoolclient/mozpoolclient.py", line 248, in close_request
    request_url = request_url + 'return/'
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

--

We likely should also break out of the retries when we get failed_bad_device
IIUC, it seems like we try to close a request for an empty URL because we never got a request_url.
http://hg.mozilla.org/build/mozharness/file/7bf07524c01b/scripts/android_panda_talos.py#l385
http://hg.mozilla.org/build/mozpool/file/f23e008c8219/mozpoolclient/mozpoolclient/mozpoolclient.py#l247

Should we just not call close_request if self.request_url is None?
http://hg.mozilla.org/build/mozharness/file/7bf07524c01b/scripts/android_panda_talos.py#l392
(Assignee)

Comment 2

3 years ago
(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #1)
> Should we just not call close_request if self.request_url is None?
> http://hg.mozilla.org/build/mozharness/file/7bf07524c01b/scripts/
> android_panda_talos.py#l392

Sounds like a simple, sane fix. Any takers?
(Assignee)

Updated

3 years ago
Assignee: nobody → coop
Status: NEW → ASSIGNED
Priority: -- → P2
(Assignee)

Comment 3

3 years ago
Created attachment 8470213 [details] [diff] [review]
[mozharness] Don't close an empty request
Attachment #8470213 - Flags: review?(jlund)
(Assignee)

Comment 4

3 years ago
Created attachment 8470214 [details] [diff] [review]
[mozpoolclient] Don't close an empty request
Attachment #8470214 - Flags: review?(dustin)
Attachment #8470214 - Flags: review?(dustin) → review+

Comment 5

3 years ago
Comment on attachment 8470213 [details] [diff] [review]
[mozharness] Don't close an empty request

Review of attachment 8470213 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, I'm guessing we should give the same love to the unittest equivalent: http://mxr.mozilla.org/build/source/mozharness/scripts/android_panda.py#472 ?
Attachment #8470213 - Flags: review?(jlund) → review+
(Assignee)

Comment 6

3 years ago
Comment on attachment 8470213 [details] [diff] [review]
[mozharness] Don't close an empty request

Review of attachment 8470213 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Jordan Lund (:jlund) from comment #5)
> Comment on attachment 8470213 [details] [diff] [review]
> [mozharness] Don't close an empty request
> 
> Review of attachment 8470213 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm, I'm guessing we should give the same love to the unittest equivalent:
> http://mxr.mozilla.org/build/source/mozharness/scripts/android_panda.py#472 ?

Landed with that additional change:

https://hg.mozilla.org/build/mozharness/rev/e29a6d6e3450
Attachment #8470213 - Flags: checked-in+
(Assignee)

Comment 7

3 years ago
Comment on attachment 8470214 [details] [diff] [review]
[mozpoolclient] Don't close an empty request

Review of attachment 8470214 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/mozpool/rev/f2d647b2c274
Attachment #8470214 - Flags: checked-in+
(Assignee)

Comment 8

3 years ago
Merged to production, and deployed [mozharness].
(Assignee)

Comment 9

3 years ago
Still need to follow the instructions to deploy a new version of the client:

https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client
(Assignee)

Comment 10

3 years ago
(In reply to Chris Cooper [:coop] from comment #9)
> Still need to follow the instructions to deploy a new version of the client:
> 
> https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client

Callek: Per these instructions, does http://hg.mozilla.org/build/tools/lib/python/vendor/mozpoolclient-0.1.4 need updating?
Flags: needinfo?(bugspam.Callek)
(Assignee)

Comment 11

3 years ago
Dustin: the new package is uploaded to pypi[1], but the package maintainer instructions aren't linked to anything in https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client. 

Are there next steps here to start using the new package?

1. http://pypi.pvt.build.mozilla.org/pub/mozpoolclient-0.1.6.tar.gz
Flags: needinfo?(dustin)
(Reporter)

Comment 12

3 years ago
(In reply to Chris Cooper [:coop] from comment #10)
> (In reply to Chris Cooper [:coop] from comment #9)
> > Still need to follow the instructions to deploy a new version of the client:
> > 
> > https://wiki.mozilla.org/ReleaseEngineering/Mozpool#Mozpool_Client
> 
> Callek: Per these instructions, does
> http://hg.mozilla.org/build/tools/lib/python/vendor/mozpoolclient-0.1.4 need
> updating?

Not sure who added me as a gating factor for that code, imo, its always worth the change here, unless its a breaking API change in which case we might have to change a few consumers in tools we should update it (after all, if it is a breaking change we can rollback with an hg backout anyway)
Flags: needinfo?(bugspam.Callek)
(Assignee)

Comment 13

3 years ago
(In reply to Justin Wood (:Callek) from comment #12)
> Not sure who added me as a gating factor for that code, imo, its always
> worth the change here, unless its a breaking API change in which case we
> might have to change a few consumers in tools we should update it (after
> all, if it is a breaking change we can rollback with an hg backout anyway)

The copy of mozpoolclient in the tools repo has been updated to 0.1.6:

https://hg.mozilla.org/build/tools/rev/056d0e0021aa
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Flags: needinfo?(dustin)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.