858865 - Android xpcshell failing with Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'

Status: RESOLVED WORKSFORME

(Testing :: XPCShell Harness, defect)

Description

[Justin Wood (:Callek)]

So, we just enabled Android XPCShell tests today, and they were promptly hidden on all trees due to high rate of intermittent failure.

error looks to TBPL like:

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-210/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:


----

The SUTAgent itself spew the following traceback:

--------- beginning of /dev/log/main
04-05 09:54:33.948 W/System.err( 1518): java.io.FileNotFoundException: /mnt/sdcard/tests/xpcshell/netwerk/test/httpserver/test/data/sjs/sjs (No such file or directory)
04-05 09:54:33.948 W/System.err( 1518): 	at org.apache.harmony.luni.platform.OSFileSystem.openImpl(Native Method)
04-05 09:54:33.948 W/System.err( 1518): 	at org.apache.harmony.luni.platform.OSFileSystem.open(OSFileSystem.java:152)
04-05 09:54:33.948 W/System.err( 1518): 	at java.io.FileOutputStream.<init>(FileOutputStream.java:97)
04-05 09:54:33.948 W/System.err( 1518): 	at java.io.FileOutputStream.<init>(FileOutputStream.java:168)
04-05 09:54:33.948 W/System.err( 1518): 	at com.mozilla.SUTAgentAndroid.service.DoCommand.Push(DoCommand.java:1960)
04-05 09:54:33.948 W/System.err( 1518): 	at com.mozilla.SUTAgentAndroid.service.DoCommand.processCommand(DoCommand.java:350)
04-05 09:54:33.948 W/System.err( 1518): 	at com.mozilla.SUTAgentAndroid.service.CmdWorkerThread.run(CmdWorkerThread.java:141)
04-05 09:54:35.628 W/System.err( 1518): java.io.IOException: Error running exec(). Command: [/data/local/xpcb/xpcw, /mnt/sdcard/tests/xpcshell/gfx/tests/unit, -r, /mnt/sdcard/tests/xpcshell/c/httpd.manifest, --greomni, /data/local/xpcb/fennec-23.0a1.en-US.android-arm-armv6.apk, -m, -n, -s, -e, const _HTTPD_JS_PATH = "/mnt/sdcard/tests/xpcshell/c/httpd.js";, -e, const _HEAD_JS_PATH = "/mnt/sdcard/tests/xpcshell/head.js";, -e, const _TESTING_MODULES_DIR = "/mnt/sdcard/tests/xpcshell/m";, -f, /mnt/sdcard/tests/xpcshell/head.js, -e, const _SERVER_ADDR = "localhost", -e, const _HEAD_FILES = [];, -e, const _TAIL_FILES = [];, -e, const _TEST_FILE = ["test_nsIScriptableRegion.js"];, -e, _execute_test(); quit(0);] Working Directory: null Environment: null
04-05 09:54:35.628 W/System.err( 1518): 	at java.lang.ProcessManager.exec(ProcessManager.java:226)
04-05 09:54:35.628 W/System.err( 1518): 	at java.lang.Runtime.exec(Runtime.java:196)
04-05 09:54:35.628 W/System.err( 1518): 	at java.lang.Runtime.exec(Runtime.java:141)
04-05 09:54:35.628 W/System.err( 1518): 	at com.mozilla.SUTAgentAndroid.service.DoCommand.StartPrg(DoCommand.java:3511)
04-05 09:54:35.628 W/System.err( 1518): 	at com.mozilla.SUTAgentAndroid.service.DoCommand.StartPrg2(DoCommand.java:3555)
04-05 09:54:35.628 W/System.err( 1518): 	at com.mozilla.SUTAgentAndroid.service.DoCommand.processCommand(DoCommand.java:689)
04-05 09:54:35.628 W/System.err( 1518): 	at com.mozilla.SUTAgentAndroid.service.CmdWorkerThread.run(CmdWorkerThread.java:141)
04-05 09:54:35.628 W/System.err( 1518): Caused by: java.io.IOException: Permission denied
04-05 09:54:35.628 W/System.err( 1518): 	at java.lang.ProcessManager.exec(Native Method)
04-05 09:54:35.628 W/System.err( 1518): 	at java.lang.ProcessManager.exec(ProcessManager.java:224)
04-05 09:54:35.628 W/System.err( 1518): 	... 6 more

(which was from http://mxr.mozilla.org/comm-central/source/mozilla/build/mobile/sutagent/android/DoCommand.java#2019 )
---------

The test harness traceback is something like

http://mxr.mozilla.org/build/source/tools/sut_tools/mozdevice/devicemanagerSUT.py#408
http://mxr.mozilla.org/build/source/tools/sut_tools/mozdevice/devicemanagerSUT.py#476
http://mxr.mozilla.org/comm-central/source/mozilla/testing/xpcshell/remotexpcshelltests.py#153

Any thoughts accepted.

Comment 1

[Geoff Brown [:gbrown]]

I think the problem may be in mkDirs, and occurs when the filename is the same as the directory that contains it. Consider http://mxr.mozilla.org/build/source/tools/sut_tools/mozdevice/devicemanager.py#116 ... and it looks like this is fixed in this version: https://hg.mozilla.org/mozilla-central/annotate/139b6ba547fa/testing/mozbase/mozdevice/mozdevice/devicemanager.py#l213

Comment 2

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=21498297&tree=Try
Android Tegra 250 try opt test xpcshell on 2013-04-05 12:07:31
slave: tegra-214

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-214/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 3

[Geoff Brown [:gbrown]]

We need the fix from bug 826492.

Comment 4

[Justin Wood (:Callek)]

Attachment: [tools] v1

Good find gbrown!

Though your example (upstream code) does make things slightly better for official (since we do the rsplit thing, it is still broken in the case of 2 dir's named the same, with last one being the repeated one)

I needed to change the one caller of dm.mkdirs in sut_tools to handle the "doesn't return anything" case properly though

This patch backports the dm.mkDirs change and fixes the sut_tools code.

I've also submitted my enumerate change as a pull request https://github.com/mozilla/mozbase/pull/47

Comment 5

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=21517894&tree=Profiling
Android Armv6 Tegra 250 profiling opt test xpcshell on 2013-04-06 16:31:08
slave: tegra-367

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-367/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 734339 [details] [diff] [review]
[tools] v1

Review of attachment 734339 [details] [diff] [review]:
-----------------------------------------------------------------

I would really really prefer to update the mozdevice code.  I don't see anything wrong with your code here.  Do we use the sut_tools devicemanager for xpcshell?  If we do, I would like us to update to the latest mozdevice

Comment 7

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=21529459&tree=Graphics
Android Armv6 Tegra 250 graphics opt test xpcshell on 2013-04-07 08:49:07
slave: tegra-286

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-286/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 8

[Geoff Brown [:gbrown]]

Comment on attachment 734339 [details] [diff] [review]
[tools] v1

Review of attachment 734339 [details] [diff] [review]:
-----------------------------------------------------------------

I'll leave it to you and :jmaher to work this out. I just want xpcshell to work reliably.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 734339 [details] [diff] [review]
[tools] v1

Review of attachment 734339 [details] [diff] [review]:
-----------------------------------------------------------------

after looking at this further, it appears we need to modify xpcshell to use the in tree devicemanager instead of the sut_tools version.  Lets land this now and fix xpcshell in the next day or two.

Comment 10

[Legacy TBPL/Treeherder Robot]

msucan
https://tbpl.mozilla.org/php/getParsedLog.php?id=21551118&tree=Try
Android Tegra 250 try opt test xpcshell on 2013-04-08 03:53:06
slave: tegra-331

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-331/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 11

[Legacy TBPL/Treeherder Robot]

bbondy
https://tbpl.mozilla.org/php/getParsedLog.php?id=21559246&tree=Try
Android Armv6 Tegra 250 try opt test xpcshell on 2013-04-08 06:59:22
slave: tegra-208

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-208/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 12

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=21561942&tree=Fx-Team
Android Tegra 250 fx-team opt test xpcshell on 2013-04-08 07:59:37
slave: tegra-218

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-218/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 13

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=21565092&tree=Graphics
Android Tegra 250 graphics opt test xpcshell on 2013-04-08 09:07:51
slave: tegra-206

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-206/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 14

[Justin Wood (:Callek)]

Comment on attachment 734339 [details] [diff] [review]
[tools] v1

http://hg.mozilla.org/build/tools/rev/3545fa27fd4e

Comment 15

[Legacy TBPL/Treeherder Robot]

Callek
https://tbpl.mozilla.org/php/getParsedLog.php?id=21590064&tree=Mozilla-Inbound
Android Tegra 250 mozilla-inbound opt test xpcshell on 2013-04-08 20:59:51
slave: tegra-217

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-217/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 16

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=21568182&tree=Graphics
Android Tegra 250 graphics opt test xpcshell on 2013-04-08 10:29:54
slave: tegra-211

Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs 72'; err='Push failed!'
Automation Error: failed making directory: /mnt
Automation Error: unable to make dirs: /mnt/sdcard/tests/xpcshell//netwerk/test/httpserver/test/data/sjs/sjs
Automation Error: Push File Failed to Validate!
TEST-UNEXPECTED-FAIL | /builds/tegra-211/test/build/tests/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test failed (with xpcshell return code: -1), see following log:

Comment 17

[William Lachance (:wlach)]

I believe this is a duplicate of bug 826492 and we're seeing this because we're not using the latest version of mozdevice. Updating xpcshell to use in-tree mozbase should fix the issue.

Comment 18

[Ed Morley [:emorley]]

(OrangeWFM bugs not modified in > 2 months)