Replay macOS recordings on linux
Categories
(Core Graveyard :: Web Replay, enhancement)
Tracking
(firefox73 fixed)
| Tracking | Status | |
|---|---|---|
| firefox73 | --- | fixed |
People
(Reporter: bhackett1024, Assigned: bhackett1024)
References
Details
Attachments
(2 files, 2 obsolete files)
As described in bug 1547081, being able to take recordings that were made on macOS and replay them on linux is a major part of the plan for cloud integration. I've looked into several ways of doing this, and I think I've found a pretty clean approach that doesn't require any changes to the firefox build system. I'd like an application targeting linux which takes as input the recording and the firefox binaries used to make the recording (plugin-container, XUL, libmozglue, libnss3, liblgpllibs, i.e. nothing that violates Apple's EULA). The application loads these binaries and links them to a set of stubs that either emulate the calls with native linux library functions (for pthreads etc.) or simply crash when being called. Functions in the latter bucket need to be redirected as part of the recording; when replaying these stubs will be redirected by the existing machinery and produce recorded information when they are called.
This design is similar to what Wine does (run windows software on linux by linking it against a set of functions emulating windows APIs). There is already a Wine-like project for running macOS software on linux, Darling (https://www.darlinghq.org/), though it doesn't support GUIs yet. The job here is much simpler than what Wine or Darling need to do, however, because we only need to emulate a small set of APIs and can rely on the recording to deal with the great bulk of the interactions between the replaying process and the system.
The main difficulty here is that macOS and linux have different binary formats (Mach-O vs. ELF). In order to load the binaries, the application needs to understand Mach-O and perform the appropriate dynamic linking steps. This seems intimidating at first but is actually not too hard and kind of fun. The Mach-O format is reasonably well documented, with open source headers and a linker implementation from Apple (APSL licensed, which is similar to BSD/MIT if you don't modify the code) and a number of third party tools for operating on binaries (none of which seem reusable here, but which are good for referencing).
The application performing this linking does not need to be compiled into firefox and its source can live in a separate place. I've started a github repository here:
https://github.com/bhackett1024/ReplayLinker
Some changes to mozilla-central/toolkit/recordreplay will be needed as well.
Disclaimer: work I'm doing on web replay is done on my own time, and is not part of my official work at mozilla. If you have a problem with me working on this, please talk to me directly.
|
Assignee | |
Comment 1•4 years ago
|
||
WIP for Gecko changes, with a fair amount of debugging crud mixed in. With this patch and the ReplayLinker repository's current code, I can record a "Hello World" page on macOS and completely replay it on Linux.
Developing this went pretty smoothly. Besides emulating the linker, the main changes needed are ways to handle the various system libraries that are not currently redirected, either by redirecting them, emulating them, or simply forwarding them to the same Linux symbol (many of these are very basic C library functions like memcpy, snprintf, and so forth). The main unexpected bit is that there are a fair number of C++ symbols linked against when using the STL, for things like std::string, std::ostream, etc. (not STL containers though, I guess because they're so extensively templatized). At first I tried emulating the behavior of these symbols, but it became too annoying so I linked against libc++.dylib in addition to the mozilla dynamic libraries. This shouldn't cause licensing issues, as libc++ is part of LLVM and has a permissive open source license.
|
|
Assignee | |
Comment 2•4 years ago
|
||
I'll also note that rr was a big help in developing this, it's nice to finally be able to use it for working on web replay!
|
|
Assignee | |
Comment 3•4 years ago
|
||
Updated patch that deals with the changes that have been going on in bug 1597149 and other connected bugs. This works and is pretty much done but I'm going to hold off on asking for review until cloud support is further along.
|
|
Assignee | |
Comment 4•4 years ago
|
||
Depends on D58436
|
|
Assignee | |
Comment 5•4 years ago
|
||
Depends on D58437
|
|
Assignee | |
Updated•4 years ago
|
Pushed by bhackett@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/cf403935cf9b Part 1 - Avoid connecting to parent when replaying remotely, r=gsvelto.
|
||
Comment 7•4 years ago
|
||
Backed out 5 changesets (bug 1603856, bug 1606447, bug 1605584, bug 1598951) for build bustages. CLOSED TREE
Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=283361477&repo=autoland&lineNumber=51406
Backout:
https://hg.mozilla.org/integration/autoland/rev/eb5a66c10085c83f5a341a7ba302c98bd6c43a2a
Pushed by bhackett@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/2e225a8de1f9 Part 1 - Avoid connecting to parent when replaying remotely, r=gsvelto.
|
||
Comment 9•4 years ago
|
||
| bugherder | ||
|
||
Comment 10•4 years ago
|
||
Pushed by bhackett@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/27a2f0bdfce9 Part 2 - Don't record random numbers in jemalloc, r=glandium.
|
||
Comment 11•4 years ago
|
||
| bugherder | ||
|
|
Assignee | |
Updated•4 years ago
|
|
||
Updated•4 years ago
|
Description
•