Closed Bug 1575008 Opened 1 year ago Closed 1 year ago

Copy https://github.com/gfx-rs/wgpu in tree and vendor its dependencies

Categories

(Core :: Graphics: WebGPU, task)

task
Not set
normal

Tracking

()

RESOLVED FIXED
mozilla72
Tracking Status
firefox72 --- fixed

People

(Reporter: kvark, Assigned: kvark)

References

Details

Attachments

(2 files)

https://github.com/gfx-rs/wgpu is a Rust library that implements WebGPU API and attempts to be both usable by the Rust applications and Gecko. The latter is achieved via "wgpu-remote" layer that manages GPU work happening in a separate process.

This task is about integrating "wgpu" code base into Gecko. Making mozilla-central to be the upstream for "wgpu" is not the only solution on the table. Alternatively we could vendor it in third-parties, but earlier conversations with :bholley and :jgilbert indicated that having m-c as the primary home for the code would allow us to iterate faster and address the issues earlier.

The main difficulty of the task is importing the third party dependencies, as well as figuring out what the path of the code is going to be.

Keywords: leave-open

Updating parking_lot would allow us to avoid dependency duplication when WebGPU tree is vendored.

Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fb816f83f64c
Update parking_lot dependency to 0.9.0 r=aosmond
Depends on: 1586291
Assignee: nobody → dmalyshau
Depends on: 1581710
Keywords: leave-open
Depends on: 1586396

This is a draft PR that vendors wgpu library in-tree and hooks up the initialization bits.
I'm still figuring out the approach to IPC (the current code use ipc_channel, which doesn't support Windows yet),
but I'm hoping to get some architectural feedback on that upcoming change.

Current architecture is roughly the following. WebGPUThreading initializes the client/server bits of wgpu "wire".
It launches a thread that waits for messages and executes them on wgpu-native. WebGPU DOM bindings would hold
"mozilla::webgpu::ffi::WGPUClient*" and use it to issue ffi calls to wgpu-remote for executing WebGPU commands.

One of the possible interesting alternative designs would be to structure wgpu-remote in such a way that
it doesn't know or care about IPC at all: it would provide some C API-wrapped basic blocks for Gecko to use
its own communication protocols, like the IPDL or Chromium thread message loops.

Attachment #9101618 - Attachment description: Include wgpu-native and wgpu-remove in the tree → Include wgpu-native and wgpu-remote in the tree

Having issues with Phabricator that struggles to handle the patch...

Depends on: 1591313
Attachment #9101618 - Attachment description: Include wgpu-native and wgpu-remote in the tree → WebGPU implementation basis
Depends on: 1298566
Depends on: 1592398
No longer depends on: 1298566
Depends on: 1591438
Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9c05fa3c0fcc
WebGPU implementation basis r=webidl,baku

Backed out for bustages on WebrtcIPCTraits.h

backout: https://hg.mozilla.org/integration/autoland/rev/6a77b82202580dd4032ef2fa54155dbffc3f57b7

push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=9c05fa3c0fcceb4f557d81e7a740bf81992aecb8&group_state=expanded&selectedJob=274540193

failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=274540193&repo=autoland&lineNumber=58684

[task 2019-11-05T04:30:45.748Z] 04:30:45 INFO - In file included from /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/webgpu/WebGPUSerialize.h:12:
[task 2019-11-05T04:30:45.749Z] 04:30:45 INFO - /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/media/webrtc/WebrtcIPCTraits.h:10:10: fatal error: 'mozilla/dom/RTCConfigurationBinding.h' file not found
[task 2019-11-05T04:30:45.749Z] 04:30:45 INFO - #include "mozilla/dom/RTCConfigurationBinding.h"
[task 2019-11-05T04:30:45.750Z] 04:30:45 INFO - ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[task 2019-11-05T04:30:45.751Z] 04:30:45 INFO - 1 error generated.
[task 2019-11-05T04:30:45.752Z] 04:30:45 INFO - /builds/worker/workspace/build/src/config/rules.mk:787: recipe for target 'UnifiedProtocols11.o' failed
[task 2019-11-05T04:30:45.753Z] 04:30:45 ERROR - make[4]: *** [UnifiedProtocols11.o] Error 1
[task 2019-11-05T04:30:45.753Z] 04:30:45 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/ipc/ipdl'
[task 2019-11-05T04:30:45.754Z] 04:30:45 INFO - make[4]: *** Waiting for unfinished jobs....
[task 2019-11-05T04:30:45.762Z] 04:30:45 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/js/xpconnect/src'
[task 2019-11-05T04:30:45.763Z] 04:30:45 INFO - js/xpconnect/src/Unified_cpp_js_xpconnect_src1.o

Flags: needinfo?(dmalyshau)

Thanks for backing out, Natalia!
It turned out that I made a simple mistake upon cleaning up and rebasing the change right before landing.
This is now addressed, and I'm trying to re-land it.

Flags: needinfo?(dmalyshau)
Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ef59d327a1a4
WebGPU implementation basis r=webidl,baku

This one doesn't look obviously related, looking into it and trying to retrigger some runs.

Flags: needinfo?(dmalyshau)

It turns out to be a legitimate issue with building SPIRV-Cross. I tracked it down and provided a fix to "cc-rs" upstream: https://github.com/alexcrichton/cc-rs/pull/455

Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/cbec8a55a1bb
WebGPU implementation basis r=webidl,baku
Depends on: 1595805

Likely will be re-landing tonight

Flags: needinfo?(dmalyshau)
Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4d16c3d62cfc
WebGPU implementation basis r=baku,bzbarsky

Backed out 2 changesets (bug 1590167, bug 1575008) for lints failures at mapped_hyph.h on a CLOSED TREE.

Backout link: Backed out 2 changesets (bug 1590167, bug 1575008) for lints failures at mapped_hyph.h on a CLOSED TREE.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=97b8c3759aae60e6513fe0c2c7e399c0814081b3&selectedJob=276060227

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=276060227&repo=autoland&lineNumber=470

Log snippet:
[task 2019-11-13T19:28:06.061Z] ./source-repo.h.stub
[task 2019-11-13T19:28:06.289Z] intl/hyphenation/glue/mapped_hyph.h.stub
[task 2019-11-13T19:28:06.289Z] Traceback (most recent call last):
[task 2019-11-13T19:28:06.289Z] File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
[task 2019-11-13T19:28:06.291Z] "main", fname, loader, pkg_name)
[task 2019-11-13T19:28:06.292Z] File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[task 2019-11-13T19:28:06.293Z] exec code in run_globals
[task 2019-11-13T19:28:06.294Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/file_generate.py", line 121, in <module>
[task 2019-11-13T19:28:06.294Z] sys.exit(main(sys.argv[1:]))
[task 2019-11-13T19:28:06.295Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main
[task 2019-11-13T19:28:06.295Z] ret = module.dict[method](output, *args.additional_arguments, **kwargs)
[task 2019-11-13T19:28:06.295Z] File "/builds/worker/checkouts/gecko/layout/style/RunCbindgen.py", line 24, in generate
[task 2019-11-13T19:28:06.295Z] env['CARGO'] = str(buildconfig.substs['CARGO'])
[task 2019-11-13T19:28:06.295Z] File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/backend/configenvironment.py", line 286, in getitem
[task 2019-11-13T19:28:06.295Z] raise KeyError("'%s'" % key)
[task 2019-11-13T19:28:06.295Z] KeyError: "'CARGO'"
[task 2019-11-13T19:28:06.296Z] backend.mk:11: recipe for target '.deps/mapped_hyph.h.stub' failed
[task 2019-11-13T19:28:06.296Z] make[4]: *** [.deps/mapped_hyph.h.stub] Error 1
[task 2019-11-13T19:28:06.296Z] /builds/worker/checkouts/gecko/config/recurse.mk:101: recipe for target 'intl/hyphenation/glue/export' failed
[task 2019-11-13T19:28:06.296Z] make[3]: *** [intl/hyphenation/glue/export] Error 2
[task 2019-11-13T19:28:06.296Z] make[3]: *** Waiting for unfinished jobs....
[task 2019-11-13T19:28:06.297Z] xpcom/xpcom-config.h.stub
[task 2019-11-13T19:28:06.297Z] build/application.ini.stub
[task 2019-11-13T19:28:06.301Z] security/apps/xpcshell.inc.stub
[task 2019-11-13T19:28:06.458Z] xpcom/xpcom-private.h.stub
[task 2019-11-13T19:28:06.466Z] security/apps/addons-public.inc.stub
[task 2019-11-13T19:28:06.468Z] build/application.ini.h.stub
[task 2019-11-13T19:28:06.476Z] security/apps/addons-public-intermediate.inc.stub
[task 2019-11-13T19:28:06.577Z] security/apps/addons-stage.inc.stub
[task 2019-11-13T19:28:06.578Z] /builds/worker/checkouts/gecko/config/recurse.mk:32: recipe for target 'export' failed
[task 2019-11-13T19:28:06.578Z] make[2]: *** [export] Error 2
[task 2019-11-13T19:28:06.578Z] /builds/worker/checkouts/gecko/config/rules.mk:389: recipe for target 'default' failed
[task 2019-11-13T19:28:06.578Z] make[1]: *** [default] Error 2
[task 2019-11-13T19:28:06.579Z] client.mk:125: recipe for target 'build' failed
[task 2019-11-13T19:28:06.579Z] make: *** [build] Error 2

Flags: needinfo?(dmalyshau)

Raul, this PR has nothing to do with "mapped_hyph.h". Was it backed out by mistake?

Flags: needinfo?(dmalyshau) → needinfo?(rgurzau)

Hi Dzmitry, i tried to backout Bug 1590167 for this failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=276060227&resultStatus=testfailed%2Cbusted%2Cexception&revision=97b8c3759aae60e6513fe0c2c7e399c0814081b3
but i got Hunks Failed in the console at this file: toolkit/library/rust/shared/Cargo.toml because this bug also modified the said file.

Flags: needinfo?(rgurzau)

ok, so it touched a file my change also touched, and git/mercurial failed to automatically resolve this...
I don't see any action I could do at this point aside from trying to re-push it.

Bug 1590167 got backed out again but when trying to push this bug i got the following conflicts:

warning: conflicts while merging .cargo/config.in! (edit, then use 'hg resolve --mark')
warning: conflicts while merging Cargo.lock! (edit, then use 'hg resolve --mark')
warning: conflicts while merging toolkit/library/rust/shared/Cargo.toml! (edit, then use 'hg resolve --mark')
warning: conflicts while merging toolkit/library/rust/shared/lib.rs! (edit, then use 'hg resolve --mark')
abort: unresolved conflicts, can't continue

Dzmitry can you take a look?

Flags: needinfo?(dmalyshau)

Let me re-land my stuff myself (via Lando). I rebased on top of autoland to reduce the chance of landing failure. Just waiting for the try to show up more - https://treeherder.mozilla.org/#/jobs?repo=try&collapsedPushes=590780&revision=41f64dd3deb7180d3661aed43088bf69bacf45bd

Flags: needinfo?(dmalyshau)
Pushed by dmalyshau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/97b72aea347c
WebGPU implementation basis r=baku,bzbarsky
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla72

For the record, the upgrade to e.g. arrayvec 0.5 (and others) should have been made in a separate patch.

Agreed, it caused much more pain that what it was worth (doing in this PR). Only went for it be cause a reviewer asked if it can be done.

You need to log in before you can comment on or make changes to this bug.