timeouts when running PSM xpcshell-test binaries on Android

RESOLVED FIXED in mozilla37

Status

()

defect
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: keeler, Assigned: Cykesiopka)

Tracking

(Depends on 1 bug, Blocks 2 bugs)

unspecified
mozilla37
ARM
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments)

See the Android failures in https://tbpl.mozilla.org/?tree=Try&rev=3fe06ac9fc7b:
14:41:36     INFO -  TEST-INFO | /builds/panda-0207/test/build/tests/xpcshell/tests/security/manager/ssl/tests/unit/test_pinning.js | running test ...
14:46:36  WARNING -  TEST-UNEXPECTED-FAIL | /builds/panda-0207/test/build/tests/xpcshell/tests/security/manager/ssl/tests/unit/test_pinning.js | Test timed out
14:46:37     INFO -  TEST-KNOWN-FAIL | /builds/panda-0207/test/build/tests/xpcshell/tests/security/manager/ssl/tests/unit/test_pinning.js | test passed (time: 300329.996ms)
I think that many of these tests time out because they are big, long, tests doing heavyweight things and the device is slow. I think we should try to increase the timeouts (requesttimeoutfactor = 4 or whatever) and re-enable them.

If there are tests that are slow because they are generating (RSA) keypairs (which is very slow), I have some ideas for how to optimize that. But, AFAICT, most of these tests are not generating keypairs during the test.

What do you think?
Flags: needinfo?(dkeeler)
I don't think any of these tests require generating keypairs. In any case, these tests work on B2G, which uses similar hardware (unless B2G xpcshell-tests only run on emulators, but in any case that would still be slow). My assumption is these tests depend on a platform-specific path that isn't correct, and when this fails, it does so in a not-immediately-fatal way, which causes the test to time out after 5 minutes. It would be great if, when an xpcshell test timed out, the log included the buffer of output from that test. I think bug 804648 aims to do this.
Depends on: 804648
Flags: needinfo?(dkeeler)
It turns out that the timeouts for these tests are caused by this:
> PROCESS_OUTPUT: Thread-145 [...] "CANNOT LINK EXECUTABLE: could not load library
> "libmozglue.so" needed by "/data/local/xpcb/OCSPStaplingServer"; caused by library
> "libmozglue.so" not found"

Note that in _setupTLSServerTest() of head_psm.js, this is done:
> 364   let greBinDir = directoryService.get("GreBinD", Ci.nsIFile);
> 365   envSvc.set("DYLD_LIBRARY_PATH", greBinDir.path);
> 366   envSvc.set("LD_LIBRARY_PATH", greBinDir.path);

... and that on Android, "GreBinD" gives something like:
>  /data/data/org.mozilla.fennec
which is not where the libs are at.
(https://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/cykesiopka.bmo@gmail.com-ea7d47149780/try-android/try_panda_android_test-xpcshell-bm100-tests1-panda-build2749.txt.gz has some logging for what the various directories that get returned are)

In addition, port 8080 on Android 4.0 is in use, so a port switch is also necessary for tests to actually pass.

I'll post the appropriate patches once the weekly pinning changes reach m-i and I confirm I haven't broken other platforms, but here's an initial try link:
  https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=675ff6489b21
Assignee: nobody → cykesiopka.bmo
Status: NEW → ASSIGNED
Depends on: 1103336
This patch makes the LD_LIBRARY_PATH change necessary to prevent the time outs, and re-enables the appropriate tests.
This patch switches the OCSP port used from 8080 to 8888 in PSM xpcshell tests to avoid port conflicts on Android 4.0.
Attachment #8528022 - Flags: review?(dkeeler)
This patch updates StaticHPKPins.h to pick up the new default-ee.der fingerprint.
Attachment #8528023 - Flags: review?(dkeeler)
Comment on attachment 8528020 [details] [diff] [review]
bug1009158_part1_LD_LIBRARY_PATH-fix_v1.patch

Try looks good so far:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=76e5d13be43f
Attachment #8528020 - Flags: review?(dkeeler)
Comment on attachment 8528020 [details] [diff] [review]
bug1009158_part1_LD_LIBRARY_PATH-fix_v1.patch

Review of attachment 8528020 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome! These should probably all be folded together into one patch when checked in.

::: security/manager/ssl/tests/unit/head_psm.js
@@ +363,5 @@
>                   .getService(Ci.nsIEnvironment);
>    let greBinDir = directoryService.get("GreBinD", Ci.nsIFile);
>    envSvc.set("DYLD_LIBRARY_PATH", greBinDir.path);
> +  // Android libraries are in /data/local/xpcb, but "GreBinD" does not return
> +  // this path on Android, so hard code it here.

Hmmm - is this a bug? Maybe it would be worth it to file a follow-up to find out/possibly fix. (Doesn't need to block this, though.)
Attachment #8528020 - Flags: review?(dkeeler) → review+
Attachment #8528022 - Flags: review?(dkeeler) → review+
Attachment #8528023 - Flags: review?(dkeeler) → review+
This is parts 1 to 3 folded together, with a minor update to part 3 to account for another weekly HPKP update.

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=775abdea824a
Attachment #8531606 - Flags: review+
Attachment #8531606 - Flags: checkin?
(In reply to David Keeler (:keeler) [use needinfo?] from comment #8)
> Hmmm - is this a bug? Maybe it would be worth it to file a follow-up to find
> out/possibly fix. (Doesn't need to block this, though.)

I'll file a follow-up once this lands.
Attachment #8531606 - Flags: checkin? → checkin+
https://hg.mozilla.org/mozilla-central/rev/7032e5b79426
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla37
Blocks: 1107794
(In reply to Cykesiopka from comment #10)
> (In reply to David Keeler (:keeler) [use needinfo?] from comment #8)
> > Hmmm - is this a bug? Maybe it would be worth it to file a follow-up to find
> > out/possibly fix. (Doesn't need to block this, though.)
> 
> I'll file a follow-up once this lands.

Filed Bug 1107794.
You need to log in before you can comment on or make changes to this bug.