1009158 - timeouts when running PSM xpcshell-test binaries on Android

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

See the Android failures in https://tbpl.mozilla.org/?tree=Try&rev=3fe06ac9fc7b:
14:41:36     INFO -  TEST-INFO | /builds/panda-0207/test/build/tests/xpcshell/tests/security/manager/ssl/tests/unit/test_pinning.js | running test ...
14:46:36  WARNING -  TEST-UNEXPECTED-FAIL | /builds/panda-0207/test/build/tests/xpcshell/tests/security/manager/ssl/tests/unit/test_pinning.js | Test timed out
14:46:37     INFO -  TEST-KNOWN-FAIL | /builds/panda-0207/test/build/tests/xpcshell/tests/security/manager/ssl/tests/unit/test_pinning.js | test passed (time: 300329.996ms)

Comment 1

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

I think that many of these tests time out because they are big, long, tests doing heavyweight things and the device is slow. I think we should try to increase the timeouts (requesttimeoutfactor = 4 or whatever) and re-enable them.

If there are tests that are slow because they are generating (RSA) keypairs (which is very slow), I have some ideas for how to optimize that. But, AFAICT, most of these tests are not generating keypairs during the test.

What do you think?

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

I don't think any of these tests require generating keypairs. In any case, these tests work on B2G, which uses similar hardware (unless B2G xpcshell-tests only run on emulators, but in any case that would still be slow). My assumption is these tests depend on a platform-specific path that isn't correct, and when this fails, it does so in a not-immediately-fatal way, which causes the test to time out after 5 minutes. It would be great if, when an xpcshell test timed out, the log included the buffer of output from that test. I think bug 804648 aims to do this.

Comment 3

[:Cykesiopka]

It turns out that the timeouts for these tests are caused by this:
> PROCESS_OUTPUT: Thread-145 [...] "CANNOT LINK EXECUTABLE: could not load library
> "libmozglue.so" needed by "/data/local/xpcb/OCSPStaplingServer"; caused by library
> "libmozglue.so" not found"

Note that in _setupTLSServerTest() of head_psm.js, this is done:
> 364   let greBinDir = directoryService.get("GreBinD", Ci.nsIFile);
> 365   envSvc.set("DYLD_LIBRARY_PATH", greBinDir.path);
> 366   envSvc.set("LD_LIBRARY_PATH", greBinDir.path);

... and that on Android, "GreBinD" gives something like:
>  /data/data/org.mozilla.fennec
which is not where the libs are at.
(https://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/[e-mail]-ea7d47149780/try-android/try_panda_android_test-xpcshell-bm100-tests1-panda-build2749.txt.gz has some logging for what the various directories that get returned are)

In addition, port 8080 on Android 4.0 is in use, so a port switch is also necessary for tests to actually pass.

I'll post the appropriate patches once the weekly pinning changes reach m-i and I confirm I haven't broken other platforms, but here's an initial try link:
  https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=675ff6489b21

Comment 4

[:Cykesiopka]

Attachment: bug1009158_part1_LD_LIBRARY_PATH-fix_v1.patch

This patch makes the LD_LIBRARY_PATH change necessary to prevent the time outs, and re-enables the appropriate tests.

Comment 5

[:Cykesiopka]

Attachment: bug1009158_part2_ocsp-8080-to-8888_v1.patch

This patch switches the OCSP port used from 8080 to 8888 in PSM xpcshell tests to avoid port conflicts on Android 4.0.

Comment 6

[:Cykesiopka]

Attachment: bug1009158_part3_update-pins-for-defaultee-fingerprint_v1.patch

This patch updates StaticHPKPins.h to pick up the new default-ee.der fingerprint.

Comment 7

[:Cykesiopka]

Comment on attachment 8528020 [details] [diff] [review]
bug1009158_part1_LD_LIBRARY_PATH-fix_v1.patch

Try looks good so far:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=76e5d13be43f

Comment 8

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Comment on attachment 8528020 [details] [diff] [review]
bug1009158_part1_LD_LIBRARY_PATH-fix_v1.patch

Review of attachment 8528020 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome! These should probably all be folded together into one patch when checked in.

::: security/manager/ssl/tests/unit/head_psm.js
@@ +363,5 @@
>                   .getService(Ci.nsIEnvironment);
>    let greBinDir = directoryService.get("GreBinD", Ci.nsIFile);
>    envSvc.set("DYLD_LIBRARY_PATH", greBinDir.path);
> +  // Android libraries are in /data/local/xpcb, but "GreBinD" does not return
> +  // this path on Android, so hard code it here.

Hmmm - is this a bug? Maybe it would be worth it to file a follow-up to find out/possibly fix. (Doesn't need to block this, though.)

Comment 9

[:Cykesiopka]

Attachment: bug1009158_combined_v1.patch

This is parts 1 to 3 folded together, with a minor update to part 3 to account for another weekly HPKP update.

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=775abdea824a

Comment 10

[:Cykesiopka]

(In reply to David Keeler (:keeler) [use needinfo?] from comment #8)
> Hmmm - is this a bug? Maybe it would be worth it to file a follow-up to find
> out/possibly fix. (Doesn't need to block this, though.)

I'll file a follow-up once this lands.

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7032e5b79426

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7032e5b79426

Comment 13

[:Cykesiopka]

(In reply to Cykesiopka from comment #10)
> (In reply to David Keeler (:keeler) [use needinfo?] from comment #8)
> > Hmmm - is this a bug? Maybe it would be worth it to file a follow-up to find
> > out/possibly fix. (Doesn't need to block this, though.)
> 
> I'll file a follow-up once this lands.

Filed Bug 1107794.