1026830 - Robocop failing to run after recent pull from m-c

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Mark Capella [:capella]]

Recently, fossterer mentioned to me his local robocop tests weren't working any longer.

I was unable to find a problem, but based on his statements that they were working up until he recently pulled and updated from m-c, I also pulled from m-c, clobbered and rebuilt both my desktop and fennec OBJDIRs and triggered a test.

Robocop now fails for me also with the same error messages:


Mochi-Remote INFO | Device info: {}
Mochi-Remote INFO | Test root: /storage/sdcard0/tests
Mochi-Remote INFO | Android sdk version '19'; will use this to filter manifests

/home/master/mozilla-central-objff/dist/bin/certutil: /usr/lib/x86_64-linux-gnu/libnss3.so: version `NSS_3.16.2' not found (required by /home/master/mozilla-central-objff/dist/bin/certutil)

TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed
Mochi-Remote ERROR | runTests() exited with code 1
Mochi-Remote ERROR | Automation Error: Exception caught while running tests

Traceback (most recent call last):
  File "_tests/testing/mochitest/runtestsremote.py", line 702, in main
    log_result = mochitest.addLogData()
  File "_tests/testing/mochitest/runtestsremote.py", line 410, in addLogData
    with open(self.localLog) as currentLog:

IOError: [Errno 2] No such file or directory: './mochitest-robocop.log'
Mochi-Remote WARNING | Unable to retrieve log file (/storage/sdcard0/tests/logs/mochitest.log) from remote device
Mochi-Remote INFO    | 

--------- beginning of /dev/log/main
06-17 20:35:47.975 W/CallerInfoCache( 1102): Previous cache task is remaining.
--------- beginning of /dev/log/system
06-17 20:35:47.975 I/ActivityManager(  606): Start proc com.google.android.talk for broadcast com.google.android.talk/com.google.android.apps.babel.realtimechat.RealTimeChatService$AlarmReceiver: pid=28970 uid=10031 gids={50031, 3003, 1028, 1015, 3002}
06-17 20:35:48.145 I/Babel   (28970): MmsConfig: mnc/mcc: 310/0
06-17 20:35:48.145 I/Babel   (28970): MmsConfig.loadMmsSettings
06-17 20:35:48.145 I/Babel   (28970): MmsConfig.loadDeviceMmsSettings from API: mUserAgent=, mUaProfUrl=
06-17 20:35:48.145 I/Babel   (28970): MmsConfig.loadFromDatabase
06-17 20:35:48.145 W/Settings(28970): Setting airplane_mode_on has moved from android.provider.Settings.System to android.provider.Settings.Global, returning read-only value.
06-17 20:35:48.155 E/Babel   (28970): canonicalizeMccMnc: invalid mccmnc 
06-17 20:35:48.155 I/Babel   (28970): MmsConfig.loadFromResources
06-17 20:35:48.155 E/Babel   (28970): canonicalizeMccMnc: invalid mccmnc nullnull
06-17 20:35:48.155 I/Babel   (28970): MmsConfig.loadMmsSettings: mUserAgent=Hangouts/2.1.100, mUaProfUrl=http://www.gstatic.com/android/hangouts/hangouts_mms_ua_profile.xml
06-17 20:35:49.046 D/GCoreUlr( 1214): Throttling analytics event: entity, activity_received, 1
06-17 20:35:50.688 W/ProcessCpuTracker(  606): Skipping unknown process pid 29038
Mochi-Remote INFO    | Device info: {}
Mochi-Remote INFO    | Test root: /storage/sdcard0/tests
INFO | runtests.py | Test summary: start.
0 INFO SimpleTest START
1 INFO TEST-START | Shutdown
2 INFO Passed: 0
3 INFO Failed: 0
4 INFO Todo: 0
5 INFO SimpleTest FINISHED
INFO | runtests.py | Test summary: end.
make: *** [mochitest-robocop] Error 1

Comment 1

[Geoff Brown [:gbrown]]

I updated from m-c and re-built my MOZ_HOST_BIN and began experiencing this problem.

It looks like NSS was updated to 3.16.2 in bug 979703 -- but that was over a month ago.

Comment 2

[Geoff Brown [:gbrown]]

It looks like the right libnss3.so is in $MOZ_HOST_BIN and can be picked up with LD_LIBRARY_PATH:

gbrown@mozpad:~/objdirs/droid$ $MOZ_HOST_BIN/certutil
/home/gbrown/objdirs/firefox/dist/bin/certutil: /usr/lib/x86_64-linux-gnu/libnss3.so: version `NSS_3.16.2' not found (required by /home/gbrown/objdirs/firefox/dist/bin/certutil)

gbrown@mozpad:~/objdirs/droid$ export LD_LIBRARY_PATH=$MOZ_HOST_BIN
gbrown@mozpad:~/objdirs/droid$ $MOZ_HOST_BIN/certutil
certutil - Utility to manipulate NSS certificate databases
...

But "make mochitest-robocop" still fails -- maybe because runtests.py does not use inherited environment to launch certutil?

Comment 3

[Geoff Brown [:gbrown]]

Here's a workaround that works for me:

Add the path pointed to by $MOZ_HOST_BIN to a new .conf file in /etc/ld.so.conf.d and then run ldconfig.

Comment 4

[Geoff Brown [:gbrown]]

:mt -- Can you offer any insight? I suppose we want to pick up the libnss3.so found in the same location as certutil (utility path / MOZ_HOST_BIN) -- is that right? What's the best way to ensure that?

Comment 5

[Martin Thomson [:mt:]]

Yeah, I've found that setting LD_LIBRARY_PATH is necessary for these tools.  It seems to be a linux-only problem: mac and windows tend to search the same folder as the binary automatically.  Thus, the following locations might be tweaked to add LD_LIBRARY_PATH=xrePath to the env:

http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#918
http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#748
http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#149

Comment 6

[Geoff Brown [:gbrown]]

Attachment: add xrePath to LD_LIBRARY_PATH

Following :mt's advice from Comment 5 works like a charm.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8442378 [details] [diff] [review]
add xrePath to LD_LIBRARY_PATH

Review of attachment 8442378 [details] [diff] [review]:
-----------------------------------------------------------------

oh, that is simple!

Comment 8

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/be39bbf27fac

Comment 9

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/be39bbf27fac