1742061 - Use NodeServers.sys.mjs to create HTTPS for xpcshell tests (where needed)

Status: RESOLVED FIXED

(Core :: Networking, task, P3)

Description

[Christoph Kerschbaumer [:ckerschb]]

We currently annotating xpcshell tests because new HttpServer() does not support https. Given that the web is migrating to http I think it's worth the investment so that our HTTPServer supports https as well - thank you!

Comment 1

[Rob Wu [:robwu]]

The "HTTPServer` used in xpcshell tests (notably extension tests) is https://searchfox.org/mozilla-central/source/netwerk/test/httpserver/httpd.js

The replacement/improvement should ideally have a compatible API.

Comment 2

[Kershaw Chang [:kershaw]]

We already have something similar here. TRRServer is a basic http2 server that only returns DNS response. We can use the same mechanism to create a generic server.

Comment 3

[Simon Friedberger [:simonf]]

Since https://bugzilla.mozilla.org/show_bug.cgi?id=1838829 has landed now. Is there a fast way to enabling HTTPS for these?

Comment 4

[Kershaw Chang [:kershaw]]

(In reply to Simon Friedberger (:simonf) from comment #3)

Since https://bugzilla.mozilla.org/show_bug.cgi?id=1838829 has landed now. Is there a fast way to enabling HTTPS for these?

Unfortunately no, because this bug is about the HTTPServer used in xpcshell tests, not mochitest.
However, we might consider use the same strategy in bug 1838829. This involves running httpd.js as a backend server and implementing an additional reverse proxy to provide HTTPS, ensuring that all existing sjs files can operate as intended and keep API compatibility.

Comment 5

[Simon Friedberger [:simonf]]

I think the ideal solution for this would be to add a reverse proxy. It means that no changes are required with respect to adding handlers in the tests. The downside would be, that adding new handlers which want to specifically react to HTTPS queries or need information about the HTTPS connection is not easily possible.

Comment 6

[Rob Wu [:robwu]]

I recently really needed to have https functionality in a unit test (bug 1990806), and tried to build upon the NodeServer.sys.mjs module introduced in bug 1988139.

I did ultimately not continue with the NodeServer.sys.mjs path, because the patch had to be uplifted to Beta where NodeServer.sys.mjs did not exist, and because of uncertainty about cross-platform support (including Android) and the fact that I would need a certificate. Instead I monkey-patched internals to downgrade https requests to http (https://searchfox.org/firefox-main/rev/21d3e8ab8b61715ddd39ac04c62a846fa79deddd/toolkit/mozapps/extensions/test/xpcshell/test_update_cn_repack_addons.js#42-70).

It would be nice if there was a way to have a reverse proxy with arbitrary domain support, or is the recommendation to use something like server-locations.txt?

For future reference, the setup with mochitests is documented at https://firefox-source-docs.mozilla.org/networking/mochitest_with_http3.html

Here is a unit test for the reverse proxy (part of bug 1809843), but it also disabled certificate verification when making the request, which is not realistic as a drop-in replacement: https://searchfox.org/firefox-main/rev/21d3e8ab8b61715ddd39ac04c62a846fa79deddd/netwerk/test/unit/test_http3_server.js#76

Comment 7

[Valentin Gosu [:valentin]]

Right now NodeServer.sys.mjs relies on having moz-http2.js running, in order to fork a new nodeJS process to run the new server - currently that's only working with xpcshell-tests, not mochitests.
But for xpc it should be possible to use NodeServer for https, h2, and proxies. The first test running outside netwerk is in test_Fetch.js.
I'll make sure to write some documentation for it.

Comment 8

[Valentin Gosu [:valentin]]

I'll keep this bug for future work to make the mochitest harness spawn moz-http2.js in order to allow NodeServers to be spawned from that.

Comment 9

[Rob Wu [:robwu]]

The bug was originally about xpcshell tests. Changing the title to "work in mochitests" seems to limit the scope to mochitests at the expense of xpcshell tests?

Could you clarify the plan to get HttpServer (which is used a lot in extension xpcshell tests) to support https?

Comment 10

[Valentin Gosu [:valentin]]

(In reply to Rob Wu [:robwu] from comment #9)

Could you clarify the plan to get HttpServer (which is used a lot in extension xpcshell tests) to support https?

I think NodeServers (NodeHTTPSServer and NodeHTTPServer) should now be good enough to use in extension xpcshell tests - I added some documentation here. Let me know if there's anything blocking you from using them.

Thanks!

Comment 11

[Rob Wu [:robwu]]

Thanks for the docs!

This bug is currently referenced by multiple xpcshell tests as the reference for HttpServer not supporting https: https://searchfox.org/firefox-main/search?q=1742061&path=&case=false&regexp=false

Could we keep this bug as the bug to track an easy way to migrate many HttpServer users in xpcshell to https? Ideally by updating the HttpServer helper (or at least createHttpServer / AddonTestUtils.createHttpServer) to support https?

Alternative if you'd like to use this bug for mochitests is to create a new bug and replace all references to this bug, with a fix to support https (or referencing a different bug number).

Comment 12

[Valentin Gosu [:valentin]]

That's a good point. I'll file a different bug for the mochitest work.
We can keep this one to start migrating some of the tests to HTTPS.

Comment 13

[Valentin Gosu [:valentin]]

Note that bug 2001895 is now fixed. NodeServer.sys.mjs should be accessible in both xpcshell and mochitests.

Comment 14

[Rob Wu [:robwu]]

I just tried to see if I can use NodeHTTPSServer in a xpcshell test. One feature gap in comparison to the createHttpServer utility in xpcshell tests is the inability to specify custom host names, such as example.com.

The helper for xpcshell tests sets up the helper like this: https://searchfox.org/firefox-main/rev/ad5f057320ecc6b934dfa1e3ec361f87712806cc/testing/modules/XPCShellContentUtils.sys.mjs#384-438

There is a bit about certificate handling for the test server at https://searchfox.org/firefox-main/rev/ad5f057320ecc6b934dfa1e3ec361f87712806cc/netwerk/docs/NodeServers.md#325-356
... but that has a hard-coded list of domains, or requires disabling of certificate validation altogether.

This may be feasible for new tests, but migrating existing tests would be tougher if every test has to do something manual for the common scenario of "start server with these host names".

Comment 15

[Valentin Gosu [:valentin]]

Attachment: Bug 1742061 - Make NodeServers.sys.mjs automatically generate a TLS cert for chosen hostnames r=#necko

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/61062bc4945c
https://hg.mozilla.org/integration/autoland/rev/475b7e4b042d
Make NodeServers.sys.mjs automatically generate a TLS cert for chosen hostnames r=necko-reviewers,kershaw

Comment 17

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/475b7e4b042d