2033393 - Upgrade dependencies for http (1.4.0), hyper (1.9/0), and warp (0.4.2) [including lots of required sub-dependencies]

Status: RESOLVED FIXED

(Firefox Build System :: General, task, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are considering to upgrade the http (0.2 -> 1.0) and warp (0.3 -> 0.4.2) crates for geckodriver given that those in-tree crates are very outdated, and external consumers of the webdriver crates run in conflicts compiling geckodriver / webdriver.

Just doing the upgrade for geckodriver / webdriver seems to work fine but the actual problem is with other in-tree components / crates that rely on these old versions, and that mach vendor rust doesn't let me vendor different versions of crates and bails out with errors:

 0:01.12 E There are 2 different versions of crate h2. Please avoid the extra duplication.
 0:01.30 E There are 2 different versions of crate headers. Please avoid the extra duplication.
 0:01.42 E There are 2 different versions of crate headers-core. Please avoid the extra duplication.
 0:01.55 E There are 2 different versions of crate http. Please avoid the extra duplication.
 0:01.67 E There are 2 different versions of crate http-body. Please avoid the extra duplication.
 0:01.80 E There are 2 different versions of crate hyper. Please avoid the extra duplication.
 0:01.92 E There are 2 different versions of crate warp. Please avoid the extra duplication.

Old crates as pulled in by:

• warp 0.3 → crashreporter (dev-dependency)
• hyper 0.14 → http3server, crashreporter (via warp 0.3)
• http 0.2 → http3server (direct), neqo-bin (external git dep), neqo-http3 (external git dep), plus transitively via hyper 0.14
• h2 0.3 → via hyper 0.14
• headers 0.3 → via warp 0.3
• headers-core 0.2 → via headers 0.3
• http-body 0.4 → via hyper 0.14

The in-tree crates to upgrade are:

• crashreporter (toolkit/crashreporter/client/app) — warp 0.3 → 0.4.2 (dev-dep, simple usage)
• http3server (netwerk/test/http3server) — hyper 0.14 → 1.x, http 0.2 → 1

The neqo-bin and neqo-http3 are external git deps (pinned at v0.26.0) that still use http 0.2. Their duplication can't be resolved without an upstream neqo upgrade or a local patch.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Lars, given the external dependencies for neqo (https://github.com/mozilla/neqo) do you think it would be possible to upgrade the dependencies as well easily or would that require more work? If we could get this done what would be the preferred way on getting it synced from your side?

CC'ing Glandium as well given that this has larger impact across different tools in our tree.

Comment 2

[Lars Eggert [:lars]]

Bumping http in neqo is easy. (We don't depend on warp.)

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That sounds good. Lars, would you be able to help with the necessary changes in neqo? I could take care of the remaining work in mozilla-central to update the crate version.

To verify that everything works end to end, I could (probably) point directly to the PR branch on GitHub and test it before you publish a new release.

Comment 4

[Lars Eggert [:lars]]

Yes!

Comment 5

[Lars Eggert [:lars]]

https://github.com/mozilla/neqo/pull/3572

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With the upstream neqo crate released I've updated my local patch and bumped related dependencies. Especially for webrender a lot of extra changes were needed and I'm not sure if those are correct. But lets see in the review. Otherwise I still have conflicts for socket2 and windows-sys that need to be solved.

Comment 7

[Nico Burns]

Cross-linking the Servo issue for this: https://github.com/servo/servo/issues/38776

The biggest blocker for us currently is icu4x 2.0. This is already upgraded in Gecko, so we'll get that automatically the next time you cut a release.

---

Regarding http/hyper/warp:

We managed to land https://github.com/servo/servo/pull/44338 and are actually mostly unblocked with regard to http crates now, with the caveat that we do currently have duplicate http crate versions in tree (caused by the webdriver crate). But as http is a small type/trait only crate that's not disasterous. We would definitely still appreciate a release with updated http version (and hyper/warp) though.

Our medium-to-long-term plan is probably (though not confirmed) to try to eliminate warp in our version of the server module of the webdriver crate and just depend on raw hyper with a custom routing layer in order to minimise dependencies. If you'd be interested in doing that then we can look at contributing that back upstream.

Either way, an upgrade to Hyper 1.0 seems like a good idea.

---

I'm surprised that WebDriver required a lot of changes. May be able to help with review there.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

To get an overview of the changes required in mozilla-central for geckodriver, as well as for many other internally used crates, please review the following data:

New crates (3)

Crate	Version
atomic-waker	1.1.2
http-body-util	0.1.3
hyper-util	0.1.20

Removed crates (1)

Crate	Version
backtrace	0.3.999

Upgraded crates (24)

Crate	Old	New
audio_thread_priority	0.34.0	0.35.1
errno	0.3.8	0.3.14
h2	0.3.26	0.4.13
headers	0.3.9	0.4.1
headers-core	0.2.0	0.3.0
http	0.2.9	1.4.0
http-body	0.4.5	1.0.1
httparse	1.8.0	1.10.1
hyper	0.14.32	1.9.0
linux-raw-sys	0.4.14	0.12.1
mio	1.1.1	1.2.0
mtu	0.3.0	0.4.1
neqo-* (7 crates)	0.26.0	0.27.0
rustix	0.38.39	1.1.4
socket2	0.5.7	0.6.3
tempfile	3.16.0	3.27.0
tokio	1.45.1	1.52.1
tokio-macros	2.5.0	2.7.0
tokio-util	0.7.2	0.7.18
warp	0.3.7	0.4.2
windows-sys	0.52.0	0.61.2

Notes

Removed: tokio-stream (0.1.12) - no longer a dependency.

The major version bumps are http (0.2->1), http-body (0.4->1), hyper (0.14->1), h2 (0.3->0.4), headers/headers-core, rustix (0.38->1), and warp (0.3->0.4).

The new crates (hyper-util, http-body-util) are the standard companion crates needed for the hyper 1.0 ecosystem.

The backtrace vendored crate was removed (the comment in Cargo.toml about it is also gone).

I still need to figure out some build issues but hopefully I should have a patch ready soon.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 2033393 - Refresh expired wildcard audits.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: WIP: Bug 2033393 - Upgrade Rust dependencies for http, hyper, and warp.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I'm not able to push the patch to try due to bug 2035588.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With bug 2035588 fixed I was able to push try builds. Hopefully the following one should be complete now so that I can ask for review:
https://treeherder.mozilla.org/jobs?repo=try&landoInstance=lando-prod-2025&landoCommitID=42647

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The following try push was all fine:
https://treeherder.mozilla.org/jobs?repo=try&revision=2abf85131583f0ffb2be7fe21d6aa3c31e2abcfb

But now I tried to rebase against mozilla-central and evertything is busted again given that lots of other updates happened in-between. It feels like an endless circle to get this upgrade landed. :(

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Looks like that I got the issues sorted out. Here a recent try build:
https://treeherder.mozilla.org/jobs?repo=try&revision=0d8c8b5e54626f3232387f7d413c386eec189b9a

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 2033393 - Vendor upgraded http, hyper, warp crates and their dependencies.

https://phabricator.services.mozilla.com/D297071

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 2033393 - Adapt consumers to http 1.x, hyper 1.x, and windows-sys HANDLE type changes.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/df3a2c86d64c
https://hg.mozilla.org/integration/autoland/rev/ecc456264b2c
Refresh expired wildcard audits. r=supply-chain-reviewers,emilio
https://github.com/mozilla-firefox/firefox/commit/f9052e80e2c8
https://hg.mozilla.org/integration/autoland/rev/f95dfdc3a655
Vendor upgraded http, hyper, warp crates and their dependencies. r=supply-chain-reviewers,webrtc-reviewers,geckodriver-reviewers,jdescottes,jgraham,bwc,valentin,afranchuk,gsvelto,gw,padenot
https://github.com/mozilla-firefox/firefox/commit/adf2e68250d3
https://hg.mozilla.org/integration/autoland/rev/4746ed3b9ded
Adapt consumers to http 1.x, hyper 1.x, and windows-sys HANDLE type changes. r=necko-reviewers,geckodriver-reviewers,gsvelto,smaug,jgraham,gw

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/d4a6e5c9effb
https://hg.mozilla.org/integration/autoland/rev/ddc486afa01f
Revert "Bug 2033393 - Adapt consumers to http 1.x, hyper 1.x, and windows-sys HANDLE type changes. r=necko-reviewers,geckodriver-reviewers,gsvelto,smaug,jgraham,gw" for causing mochitest failures.

Comment 19

[agoloman]

Backed out for causing mochitest failures.

• Backout link
• Push with failures
• Failure Log
• Failure Log

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The mochitests for network are failing due to a regression introduced by my patch. I'll push an update which will fix it.

The gl1c mochitests are failing because the manifest files only contain fail-if entries for macOS 15.3 but some of the machines still run on macOS 15.2 (24C101). Ryan, is there a reason why not all workers run 15.3? It's a huge list of those toml entries:

https://searchfox.org/firefox-main/search?q=os_version+%3D%3D+%2715.&path=toml&case=false&regexp=false

Comment 21

[Ryan Curran]

thanks for the heads the up. these were new workers should've had 15.3 on them before going live. the workers have been quarantined, updated, and put back into prod

Comment 22

[Oskar Mansfeld]

We have vendored Neqo to v0.28.0 with the old http version in Bug 2039361 and now did another Release v0.28.1 that just contains the bump to http@1.4.0.

That one should now be used for this effort here.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thank you Oskar for the details to which version I have to update. I'm doing the rebase now and will most likely push a couple of jobs to try first before moving forward with the landing later today after the merge to mozilla-beta.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I pushed a new try build with some previously failing tests to ensure it now works all fine:
https://treeherder.mozilla.org/jobs?repo=try&landoInstance=lando-prod-2025&landoCommitID=47458

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/2c666359820b
https://hg.mozilla.org/integration/autoland/rev/c1be7a11152e
Refresh expired wildcard audits. r=supply-chain-reviewers,emilio
https://github.com/mozilla-firefox/firefox/commit/0ba394a1ccb2
https://hg.mozilla.org/integration/autoland/rev/778460935d14
Vendor upgraded http, hyper, warp crates and their dependencies. r=supply-chain-reviewers,webrtc-reviewers,geckodriver-reviewers,jdescottes,jgraham,bwc,valentin,afranchuk,gsvelto,gw,padenot
https://github.com/mozilla-firefox/firefox/commit/8bb1110a8850
https://hg.mozilla.org/integration/autoland/rev/5cc939f03130
Adapt consumers to http 1.x, hyper 1.x, and windows-sys HANDLE type changes. r=necko-reviewers,geckodriver-reviewers,gsvelto,smaug,jgraham,gw,kershaw

Comment 26

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/c1be7a11152e
https://hg.mozilla.org/mozilla-central/rev/778460935d14
https://hg.mozilla.org/mozilla-central/rev/5cc939f03130