1608126 - Node.js is not available when running xpcshell tests on Fenix, making HTTP/2 tests fail

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect, P2)

Description

[Honza Bambas (:mayhemer)]

Please see the following try push: https://treeherder.mozilla.org/#/jobs?repo=try&collapsedPushes=511147%2C509985%2C511151&selectedJob=284192682&revision=0c7c19c3b79400b49893c9b97d0da495c49af817

Only mention of "node" in the full raw log is:

[task 2020-01-08T14:35:35.973Z] 14:35:35     INFO -  MOZ_NODE_PATH environment variable not set. Tests requiring http/2 will fail.

Comment 1

[Geoff Brown [:gbrown]]

afaik, we have never tried to run node for Android tests. I think it would be fairly easy to configure node to run on the host for Android tests, but I don't know what's involved in getting tests running on the Android device to communicate with node running on the host.

Comment 2

[Honza Bambas (:mayhemer)]

Thanks Geoff. Can we use this bug to drive that effort? Fenix is now a tier-1 product and there is a plenty of tests we disable on Android (formerly Fennec, now Fenix). We want to enable them (and possibly also fix, if necessary) from obvious reasons.

Comment 3

[Geoff Brown [:gbrown]]

Sure. Here's a good start:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=053ae1a4ff35d5fe2e1e779d9e24d2da627b38ba

I think node is being started on the host and MOZHTTP2_PORT/MOZNODE_EXEC_PORT are being set correctly, but we still need to get the device talking to the host. Maybe I can use emulator port forwarding? I can try that next week...

Comment 4

[Geoff Brown [:gbrown]]

I realize now that emulator port forwarding is not a good idea: We want to be able to run xpcshell tests on physical devices as well.

Is there a way to set up a proxy or something to link up the tests, running on device, to the node server running on the host?

The patch in the try push of comment 3 demonstrates how we can run node on the host, but I'm not sure how to proceed from there.

Comment 5

[Honza Bambas (:mayhemer)]

Geoff, thanks for the attempt to fix this in comment 3!

Sorry to mess with priorities on your behalf, but this blocks a P2 bug 1567097 affecting a Tier-1 platform.

I can't answer any of your questions in comment 3 and 4. Someone from the test infra team needs to look at this. Do you know about anyone to forward this to? I can definitely provide some (limited) help if needed, too.

Thanks.

Comment 6

[Geoff Brown [:gbrown]]

I don't dispute the importance or priority of the bug, but there's still no one assigned. I don't think the pressing issues are test infra - we need someone who understands the HTTP/2 tests to describe what's needed.

:valentin - I see you have commented on some other Android HTTP/2 test bugs; do you have a plan for how this should work? I can run node on the (Linux) test host, but I'm not sure if that's helpful or appropriate -- I don't know how/if the tests on devices can/should communicate with node on the host. If that's not an appropriate network architecture, do we need to run node on Android?

Comment 7

[Valentin Gosu [:valentin] (he/him)]

Hi Geoff,
So the plan here is to run this only for the x64 emulator. nodejs would be running on the host - and after running it we somehow pass the env-variables to Fenix running in the emulator, which will then connect to the ports running on the host via TCP, right?
I think this would be a good enough strategy to exercise the android code paths for HTTP2 - even if it's not as good as running node directly on the device.

I think most tests should work in this configuration if we have two way TCP connectivity between host and emulator (which seems like there is: https://developer.android.com/studio/run/emulator-networking )
We do have to make some changes to some of the tests that assume node will always be on localhost - for example here we'd have to point it to 10.0.2.2 instead of 127.0.0.1
I can try to fixup all these issues, on the condition that the try patch works locally with the emulator.

Comment 8

[Honza Bambas (:mayhemer)]

Looks like this is related to an existing bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1567342

Comment 9

[Geoff Brown [:gbrown]]

Thanks Valentin!

Minor issue: If we run only on emulator, we should mark the relevant tests as skipped on (android && !is_emulator), to avoid confusion if someone tries to run against hardware -- I'll add support for is_emulator in bug 1609723.

Using 10.0.2.2 instead of 127.0.0.1 should work fine. If there is a fixed set of ports involved, another possibility is using adb port forwarding (adb forward/reverse commands); that could be implemented entirely in the harness. Do we just need to worry about MOZHTTP2_PORT and MOZNODE_EXEC_PORT, or are there other ports involved?

Comment 10

[Geoff Brown [:gbrown]]

It looks like adb port forwarding may work:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=285235199&repo=try&lineNumber=2049

[task 2020-01-16T18:26:44.788Z] 18:26:44     INFO -  TEST-START | netwerk/test/unit/test_http2.js
[task 2020-01-16T18:27:00.642Z] 18:27:00     INFO -  TEST-PASS | netwerk/test/unit/test_http2.js | took 15854ms

Comment 11

[Geoff Brown [:gbrown]]

Yes, looks good:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0a8672a118356184545ab8fd5a4e946e842a3cc6

and that will work for devices too - no need for is_emulator annotations.

Comment 12

[Geoff Brown [:gbrown]]

Attachment: Bug 1608126 - Enable node for Android xpcshell tests and run node tests on Android; r=

Fetch, configure, and run node for Android on the test host, just like Linux tests do.
Make the node/HTTP/2 environment variables available to the tests on the device, and
use adb port forwarding to connect sockets. Finally, enable tests skipped for node.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11a73f4bbb1a
Enable node for Android xpcshell tests and run node tests on Android; r=valentin

Comment 14

[Narcis Beleuzu [:NarcisB]]

Backed out for bustages on runxpcshelltests.py

Backout link: https://hg.mozilla.org/integration/autoland/rev/06b1d6b3800216a1d8ca31db900f89f702a70d78
Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=285553741&repo=autoland&lineNumber=56379

Comment 15

[Narcis Beleuzu [:NarcisB]]

There were also xpcshell failures like https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=285554410&repo=autoland&lineNumber=889

Comment 16

[Geoff Brown [:gbrown]]

I was sure my changes would not affect desktop; I should have checked on try!

Comment 17

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/5314374fd2c0

Comment 18

[Honza Bambas (:mayhemer)]

Thank you guys!