Closed Bug 1526752 Opened 5 years ago Closed 5 years ago

[mozdevice] ADBError: cannot use a string pattern on a bytes-like object: adb is not executable

Categories

(Testing :: Mozbase, defect, P3)

Product:
Testing
Component:
Mozbase
Version:
Version 3
Type:
defect

Tracking

(firefox69 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla69
Tracking Flags:
Tracking Status
firefox69 --- fixed

People

(Reporter: davehunt, Assigned: marauder)

References

Details

Attachments

(1 file)

Bug 1526752 - [mozdevice] Update adb.py and other necessary files to support python3 r?ahal
47 bytes, text/x-phabricator-request
Details | Review

Running the following code under Python 3 throws an exception:

from mozdevice import ADBAndroid
device = ADBAndroid()

The traceback of the exception is:

Traceback (most recent call last):
  File "/Users/dhunt/.local/share/virtualenvs/raptor-studio-X1UPyeBo/lib/python3.7/site-packages/mozdevice/adb.py", line 192, in __init__
    self._adb_version = re_version.match(output[0]).group(1)
TypeError: cannot use a string pattern on a bytes-like object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dhunt/.local/share/virtualenvs/raptor-studio-X1UPyeBo/lib/python3.7/site-packages/mozdevice/adb.py", line 601, in __init__
    require_root=require_root)
  File "/Users/dhunt/.local/share/virtualenvs/raptor-studio-X1UPyeBo/lib/python3.7/site-packages/mozdevice/adb.py", line 195, in __init__
    raise ADBError('%s: %s is not executable.' % (exc, adb))
mozdevice.adb.ADBError: cannot use a string pattern on a bytes-like object: adb is not executable.

Improve support for Python 3 by running modernize over adb.py and a few other related changes.

Assignee: nobody → dave.hunt
Status: NEW → ASSIGNED

I believe some of the changes here will only affect the geckoview-junit test, so it would be great to see those tests verified on try.

Yes, that looks fine. Thank you!

Pushed by dhunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a465a9a1a913
[mozdevice] Run modernize over adb.py; r=ahal

Backed out changeset a465a9a1a913 (bug 1526752) for mochitest failures with ValueError: need more than 1 value to unpack

Backout: https://hg.mozilla.org/integration/autoland/rev/f17b109fc6acc85d91cbac9b566b696d26052cd9

Failure push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=a465a9a1a91318af3f4a13adae4feaec3bb373a9&selectedJob=229892401

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=229892401&repo=autoland&lineNumber=1898

task 2019-02-22T13:17:29.419Z] 13:17:29 INFO - Traceback (most recent call last):
[task 2019-02-22T13:17:29.419Z] 13:17:29 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 2798, in doTests
[task 2019-02-22T13:17:29.420Z] 13:17:29 INFO - e10s=options.e10s
[task 2019-02-22T13:17:29.420Z] 13:17:29 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 341, in runApp
[task 2019-02-22T13:17:29.420Z] 13:17:29 INFO - ret, _ = self.automation.runApp(*args, **kwargs)
[task 2019-02-22T13:17:29.421Z] 13:17:29 INFO - File "/builds/worker/workspace/build/tests/mochitest/remoteautomation.py", line 65, in runApp
[task 2019-02-22T13:17:29.421Z] 13:17:29 INFO - e10s=e10s, **self.processArgs)
[task 2019-02-22T13:17:29.421Z] 13:17:29 INFO - File "/builds/worker/workspace/build/tests/mochitest/remoteautomation.py", line 299, in launchApp
[task 2019-02-22T13:17:29.421Z] 13:17:29 INFO - self.device.launch_fennec(self.appName, moz_env=env, extra_args=args, url=url)
[task 2019-02-22T13:17:29.421Z] 13:17:29 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 3067, in launch_fennec
[task 2019-02-22T13:17:29.422Z] 13:17:29 INFO - for (env_count, (env_key, env_val)) in moz_env.items():
[task 2019-02-22T13:17:29.422Z] 13:17:29 INFO - ValueError: need more than 1 value to unpack
[task 2019-02-22T13:17:29.423Z] 13:17:29 INFO - 0 ERROR Automation Error: Received unexpected exception while running application
[task 2019-02-22T13:17:29.423Z] 13:17:29 INFO - Stopping web server
[task 2019-02-22T13:17:29.435Z] 13:17:29 INFO - Stopping web socket server
[task 2019-02-22T13:17:29.455Z] 13:17:29 INFO - Stopping ssltunnel
[task 2019-02-22T13:17:29.476Z] 13:17:29 INFO - leakcheck | refcount logging is off, so leaks can't be detected!
[task 2019-02-22T13:17:29.477Z] 13:17:29 INFO - runtests.py | Running tests: end.
[task 2019-02-22T13:17:30.109Z] 13:17:30 INFO - Buffered messages finished
[task 2019-02-22T13:17:30.110Z] 13:17:30 INFO - Running manifest: dom/console/tests/mochitest.ini
[task 2019-02-22T13:17:31.431Z] 13:17:31 INFO - adb Ignoring attempt to chmod external storage
[task 2019-02-22T13:17:31.456Z] 13:17:31 INFO - runtests.py | Failed to copy /builds/worker/workspace/build/tests/mochitest/hyphenation to profile
[task 2019-02-22T13:17:31.522Z] 13:17:31 INFO - pk12util: PKCS12 IMPORT SUCCESSFUL
[task 2019-02-22T13:17:31.652Z] 13:17:31 INFO - MochitestServer : launching [u'/builds/worker/workspace/build/hostutils/host-utils-66.0a1.en-US.linux-x86_64/xpcshell', '-g', '/builds/worker/workspace/build/hostutils/host-utils-66.0a1.en-US.linux-x86_64', '-f', '/builds/worker/workspace/build/hostutils/host-utils-66.0a1.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpkf6L3m.mozrunner'; const _SERVER_PORT = '8854'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/builds/worker/workspace/build/tests/mochitest/server.js']
[task 2019-02-22T13:17:31.652Z] 13:17:31 INFO - runtests.py | Server pid: 1121
[task 2019-02-22T13:17:31.656Z] 13:17:31 INFO - runtests.py | Websocket server pid: 1124
[task 2019-02-22T13:17:31.663Z] 13:17:31 INFO - runtests.py | SSL tunnel pid: 1127
[task 2019-02-22T13:17:35.506Z] 13:17:35 INFO - adb Ignoring attempt to chmod external storage
[task 2019-02-22T13:17:35.507Z] 13:17:35 INFO - runtests.py | Running with scheme: http
[task 2019-02-22T13:17:35.507Z] 13:17:35 INFO - runtests.py | Running with e10s: False
[task 2019-02-22T13:17:35.507Z] 13:17:35 INFO - runtests.py | Running with serviceworker_e10s: False
[task 2019-02-22T13:17:35.507Z] 13:17:35 INFO - runtests.py | Running tests: start.
[task 2019-02-22T13:17:35.716Z] 13:17:35 INFO - Traceback (most recent call last):
[task 2019-02-22T13:17:35.717Z] 13:17:35 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 2798, in doTests
[task 2019-02-22T13:17:35.717Z] 13:17:35 INFO - e10s=options.e10s
[task 2019-02-22T13:17:35.717Z] 13:17:35 INFO - File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 341, in runApp
[task 2019-02-22T13:17:35.717Z] 13:17:35 INFO - ret, _ = self.automation.runApp(*args, **kwargs)
[task 2019-02-22T13:17:35.718Z] 13:17:35 INFO - File "/builds/worker/workspace/build/tests/mochitest/remoteautomation.py", line 65, in runApp
[task 2019-02-22T13:17:35.718Z] 13:17:35 INFO - e10s=e10s, **self.processArgs)
[task 2019-02-22T13:17:35.718Z] 13:17:35 INFO - File "/builds/worker/workspace/build/tests/mochitest/remoteautomation.py", line 299, in launchApp
[task 2019-02-22T13:17:35.718Z] 13:17:35 INFO - self.device.launch_fennec(self.appName, moz_env=env, extra_args=args, url=url)
[task 2019-02-22T13:17:35.719Z] 13:17:35 INFO - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 3067, in launch_fennec
[task 2019-02-22T13:17:35.719Z] 13:17:35 INFO - for (env_count, (env_key, env_val)) in moz_env.items():
[task 2019-02-22T13:17:35.719Z] 13:17:35 INFO - ValueError: need more than 1 value to unpack
[task 2019-02-22T13:17:35.719Z] 13:17:35 INFO - 1 ERROR Automation Error: Received unexpected exception while running application

Flags: needinfo?(dave.hunt)

Please do a full android test run on try ./mach try fuzzy --full --query 'android' for these type of changes on adb.py.

I believe this is due to a missing enumerate() call on this line: https://hg.mozilla.org/integration/autoland/rev/a465a9a1a913#l1.260 but I haven't had the bandwidth to look into this.

Marian or Alex: Would you be able to pick this up? With this resolved we should be able to convert Raptor studio to a Python 3 project.

Flags: needinfo?(marian.raiciof)
Flags: needinfo?(dave.hunt)
Flags: needinfo?(alexandru.ionescu)

Dave, talked to Marian and he said that he will handle this. Did he?

Flags: needinfo?(alexandru.ionescu)

I have made the change in adb.py and ran a push to try:

./mach try fuzzy --full --query 'android'

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0e3322b76799e7bfa92c0b844c560a2bb451bcaf

Flags: needinfo?(marian.raiciof)

I just landed Bug 1525401 on autoland and bumped the version to 3.0.2. Could you go ahead and publish to pypi when you land this one? Thanks.

:marauder could you rebase and push to try again? your push has a lot of failures, which I don't believe are related to this change so I'm wondering if a rebase would help.

Flags: needinfo?(marian.raiciof)

It seems likely that the failures are related to the changes here:

[task 2019-04-03T12:18:10.678Z] 12:18:10 INFO - remoteautomation.py | exception reading log: 'utf8' codec can't decode byte 0xb1 in position 0: invalid start byte

(In reply to Geoff Brown [:gbrown] from comment #15)

It seems likely that the failures are related to the changes here:

[task 2019-04-03T12:18:10.678Z] 12:18:10 INFO - remoteautomation.py | exception reading log: 'utf8' codec can't decode byte 0xb1 in position 0: invalid start byte

Thanks :gbrown I completely missed that!

I saw that error from remoteautomation.py and is triggered from line 258:
newLogContent = self.device.get_file(self.remoteLog, offset=self.stdoutlen)

and catched on line 262:
self.log.info("remoteautomation.py | exception reading log: %s" % str(e))

When i investigated this issue, i found that the remoteLog is used in
mozilla-central/testing/mochitest/runtestsremote.py
line 51:
self.remoteLogFile = posixpath.join(options.remoteTestRoot, "logs", "mochitest.log")
line 63:
self.automation = RemoteAutomation(self.device, options.remoteappname, self.remoteProfile,
self.remoteLogFile, processArgs=process_args)

Is there any documentation about the method get_file from self.device.get_file() ?
(Update: I think i found it - is in adb.py - line 2250)
(Update2: note to myself -
To try :
import io
f = io.open("test", mode="r", encoding="utf-8")
)

I assumed that the remoteLog will always be in bytes since is a file - so decoding should work on it,
but these "solutions" didn't work:

newLogContent = self.device.get_file(self.remoteLog.decode('utf-8', 'ignore'), offset=self.stdoutlen)
newLogContent = self.device.get_file(self.remoteLog.decode('utf-8', 'replace'), offset=self.stdoutlen)

newLogContent = self.device.get_file(self.remoteLog, offset=self.stdoutlen).decode('utf-8', 'ignore')
newLogContent = self.device.get_file(self.remoteLog, offset=self.stdoutlen).decode('utf-8', 'replace')

Regards,
Marian

Flags: needinfo?(marian.raiciof)

Another push to try (the first one did not display the Android environments):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=48bc4e32ac6000d42e14d898048d7c2d94fbb11f

(In reply to Marian Raiciof [:marauder] from comment #19)

There are a lot of failures there, but some -- cppunit, gtest -- are expected with artifact builds, and most of the others are tier 2 and look unrelated to the changes: I think there are no concerns.

Hi Geoff,

Yes, I also looked into logs for failing tests and most of them are cold tests that are running more than 30 minutes and fail because of "Task aborted - max run time exceeded".
Some tp6m tests are failing because of "raptor_script.py: error: no such option: --disable-e10s" - a bug was opened for this and a solution was provided.
Other tests are failing because of intermittent failures - timeouts. We already have open bugs for those.

Priority: -- → P3
Attachment #9042819 - Attachment is obsolete: true
Attachment #9042819 - Attachment is obsolete: false
Assignee: dave.hunt → marian.raiciof
Attachment #9042819 - Attachment description: Bug 1526752 - [mozdevice] Run modernize over adb.py; r?ahal → Bug 1526752 - [mozdevice] Update adb.py and other necessary files to support python3 r?ahal
Pushed by mraiciof@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/36c3d348039e
[mozdevice] Update adb.py and other necessary files to support python3 r=ahal,davehunt

https://hg.mozilla.org/mozilla-central/rev/36c3d348039e

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox69: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla69
Regressions: 1560339
See Also: → 1642672
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: