Closed Bug 1661539 Opened 6 years ago Closed 5 years ago

iOS megazord: Build A-S and Glean as single binary

Categories

(Data Platform and Tools :: Glean: SDK, task, P1)

Product:
Data Platform and Tools
Component:
Glean: SDK
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: garvan, Assigned: janerik)

References

Details

Attachments

(1 file)

GitHub Pull Request
57 bytes, text/x-github-pull-request
Details | Review

Having Glean as a separate binary from A-S we are seeing:

A single binary seems clearly the answer here for reduced product risk in terms of startup cost, binary size cost, risk of crashes, and reduced iOS eng time costs.

I still think the stack behavior is not necessarily a fault of having 2 Rust libraries in there. This could just as well happen for any combination of Swift, C and Rust. We should figure out the current situation on its own.

We had the discussion of what a-s calls "megazording" a long time ago and so far agreed to not do it, because it has overhead on both teams to maintain that and complicates how we release things.
Now this was in the context of Android mostly.

I think a megazording for iOS might be more achievable and could even live outside of a single repository (either one of our repos doing it or a third repository only handling the megazording through submodules or similar).
IMO we should set up another discussion around this to find a way forward.

(I disagree with the points regarding CI. Even with a single megazorded release we will continue to have individual CI machines, because those projects run independently and we need to test them individually. But this is also something the iOS engineers shouldn't need to deal with)

To tackle this I manually built a megazord mix from appservices and Glean.

Megazord'ed appservices/Glean fork: https://github.com/mozilla/application-services/compare/main...badboy:glean-megazord-hack
Firefox-iOS using that fork: https://github.com/mozilla-mobile/firefox-ios/compare/main-dev...badboy:megazord-appservices-glean

Building locally using Xcode 12 beta5. Running it in the simulator (iPhone 11, iOS 14).
Glean's working: https://debug-ping-preview.firebaseapp.com/pings/jer-megazord (sample pings)
I haven't checked the binary size and/or startup time impact mostly because I have no idea how to measure that.

This is purely for testing!
To bring that into a maintainable state we would need to find a way to either be a submodule in a-s or (I favor this:) put the megazord into its own repository.
We will probably require some shell scripting anyway to make it all play nice together (luckily glean-ffi is pushed to crates.io, so that part could be pulled down from there)

Megazord'ed appservices/Glean fork: https://github.com/mozilla/application-services/compare/main...badboy:glean-megazord-hack
Firefox-iOS using that fork: https://github.com/mozilla-mobile/firefox-ios/compare/main-dev...badboy:megazord-appservices-glean
This is purely for testing!

Thanks this looks great. I don't think it should be testing-only, this is a better solution than our current workaround, which may only workaround that known stack corruption and not some other edge case.

I haven't checked the binary size and/or startup time impact mostly because I have no idea how to measure that.

np, I will handle those investigations later on, but I am confident we will see improvements there with the megazord.

It looks like doing a megazord for iOS would solve multiple problems at once: statically linking of Glean, overall size impact (no more libstd duplication) & the weird stack misalignment issue mentioned.

The above PoC shows that megazording on iOS is simpler than it will be on Android.
Firefox iOS has a very short timeline for the next release, based on the iOS 14 release in mid September.
At a minimum the stack misalignment needs to be fixed until then.

We can do all at once.
Here’s a short proposal on the next steps:

Step 1 -- Move beyond a PoC

Do a megazord release through application-services.
It can depend on the crates.io release of glean-ffi.
It can either:

  • submodule mozilla/glean (pinned to release version), then add it to the Xcode project for full compilation
  • or have some shell scripts to check out only glean-core/ios/Glean when building a release.

The Glean team is responsible to ensure matching versions of glean-ffi and the Swift code, a quick update requires a Pull Request on mozilla/application-services.
The application-services team is responsible for a release.
Firefox iOS consumed the MozillaAppServices.framework.zip as usual from a release, doesn’t consume Glean.framework.zip (minimal changes to how it generates metrics code required).

Timeline: 2 weeks.
Who: the Glean team can work it out and send a PR, a-s team does a release.

Step 2 -- Do it properly

Move the megazord part into a separate repository, used for releases only.
This will require more shell scripting to fit it all together, but both teams can independently push releases.
What’s in this extra repository is considered stable and ready for release.
The produced MozillaASGlean.framework.zip (name bikeshedable) replaces both MozillaAppServices.framework.zip and Glean.framework.zip in consumers such as Firefox iOS

Timeline: 4 weeks? Not sure. I don’t want this to linger around for too long.
Who: ?

I'm open to suggestions for doing this differently.
It would be good if we can make a decision about the way forward this week.
This requires approval/buy-in from both teams.

It would be good if we can make a decision about the way forward this week.

I'll surface this in team discussion this week, although I believe a few folks may be PTO.

put the megazord into its own repository.

I also favour this approach FWIW.

In my ideal world, we would provide sufficiently-advanced tooling that the Firefox iOS team can manage this themselves at low cost, perhaps in a dedicated firefox-ios-megazord repository or even inline in the main firefox-ios repository. There are practical reasons why that's not tenable in the short term, but I think it's interesting to consider whether a world where each project has its own megazord could exist and what we'd need in order to get there.

two sets of additional CI machines to work with rather than one additonal

Echoing Comment 1, I don't understand what you mean here in terms of cost/complexity to Firefox iOS specifically (or, is it a more general cost/complexity to Mozilla as a whole?).

(In reply to Ryan Kelly [:rfkelly] from comment #6)

two sets of additional CI machines to work with rather than one additional

Echoing Comment 1, I don't understand what you mean here in terms of cost/complexity to Firefox iOS specifically (or, is it a more general cost/complexity to Mozilla as a whole?).

Right, will clarify, LMK if you need more info on those points I listed.

For Firefox iOS. Every additional build system we depend on is an added pain point because we need binary compat, and these CIs need to be synchronized to fx-ios needs for this; fx-ios cycles (either release cycles or Xcode update cycles) don't align with the dependent libs. We have the added load across teams of trying to synchronize on that. We have been through 6 XCode releases in 4 months, and we didn't even make an effort to try keep CIs updated for this due to the constant workload of that, and just built locally. The PRs per-CI system update can be rather heavy (see https://github.com/mozilla/glean/commit/810cefc47bf0f22a5e1acca62da18e8f41df4b34 for instance), so there is 2x cognitive load to keep updating these machines for fx-ios needs.
The surface area of potential problems with our integrations with those CI machines is increased by having two systems. The communication complexity is increased with two different teams to communicate with about CI issues.
In summary, it 'feels' like 2x the systems, 2x the problems, but that is likely an exaggeration based on the newness of integrating these two systems.

Thanks, that clarifies things a lot!

Suppose we had a world where the combined megazord lives in its own repository, firefox-ios-megazord. I wonder how much of the above work could be done once in the firefox-ios-megazord repo, perhaps then propagating back down into appservices and glean in an easier manner.

(In reply to Ryan Kelly [:rfkelly] from comment #5)

I'll surface this in team discussion this week, although I believe a few folks may be PTO.

Right, short week and everything. Surfacing that in teams is good enough then, we can reach a decision next week.

Do a megazord release through application-services.
It can depend on the crates.io release of glean-ffi.
It can either:
submodule mozilla/glean (pinned to release version), then add it to the Xcode project for full compilation

We discussed this birefly in the Sync team meeting earlier today. Mark and Lina weren't there so I'm ni?ing them here to please weigh in with any additional comments or concerns.

Broadly, we're open to this plan of importing glean into the iOS megazord as a git submodule, particularly if the work is being driven by the Glean team. We should keep the conversation going about how to iterate towards a better longer-term setup, it seems potentially valuable enough to try in the short-term.

One possible downside of this approach: if we land a breaking API change in appservices, and glean publishes a new release, then it will be hard for Firefox iOS to pick up the new glean version without also updating to the breaking appservices change. This is probably manageable in practice but is worth being aware of. (And we have the same problem with other components in the appservices megazord right now, e.g. you can't pick up a backwards-compatible change to bookmarks if it's in the same release as a breaking change to FxA).

What should we expect in terms of release/update frequency from the Glean team? (Just to set expectations about how it would flow through to our own review and release workload).

Flags: needinfo?(markh)
Flags: needinfo?(lina)

This sounds fine to me. There will be alot of devil in the details, but that's just the nature of complex systems. We already face similar problems with app-services rust code in mozilla-central - there's only a tiny amount there now and already we have faced the "how do we manage the breaking changes?" scenario mentioned - and we expect the amount of rust code in mozilla-central to grow significantly, so these problems to also grow significantly, then we have multipliers like this and our heads will explode.

For this reason, I'd tend to avoid suggesting we will grow the perfect solution in the first instance. There's nothing wrong with doing some slightly hacky things while we understand the problem space a bit better. However, it seems like the right people are on the case here, so 👍 from me.

Flags: needinfo?(markh)

(In reply to Ryan Kelly [:rfkelly] from comment #10)

Broadly, we're open to this plan of importing glean into the iOS megazord as a git submodule, particularly if the work is being driven by the Glean team. We should keep the conversation going about how to iterate towards a better longer-term setup, it seems potentially valuable enough to try in the short-term.

👍

One possible downside of this approach: if we land a breaking API change in appservices, and glean publishes a new release, then it will be hard for Firefox iOS to pick up the new glean version without also updating to the breaking appservices change. This is probably manageable in practice but is worth being aware of. (And we have the same problem with other components in the appservices megazord right now, e.g. you can't pick up a backwards-compatible change to bookmarks if it's in the same release as a breaking change to FxA).

This is already true for Android Components, through which Glean is shipped. It can be annoying for consumers, but so far it hasn't been a big issue (and the reverse is also true: Glean could do breaking changes)

What should we expect in terms of release/update frequency from the Glean team? (Just to set expectations about how it would flow through to our own review and release workload).

We don't have a regular release schedule for Glean. It's mostly driven by feature requirements and seldom bug fixes.
I don't think anything quicker than 2 weeks is expected.
What overhead for releases do you expect to have by bundling Glean?

What overhead for releases do you expect to have by bundling Glean?

Not a lot, I don't think - IIUC the proposal is that the Glean team would make a PR to update appservices with the new release, and we just have to coordinate ourselves to make a release in turn. In theory this should be fine because in theory we should be OK to cut a release from main at any stage...so I think we can safely try it out and see.

Alright. Unless I here any objections I plan to give this a first try next week!

Assignee: nobody → jrediger
Type: defect → task
Priority: -- → P1
Summary: iOS: Build A-S and Glean as single binary to avoid stack corruption risk and binary bloat → iOS megazord: Build A-S and Glean as single binary
Attached file GitHub Pull Request —

Went ahead and tried to integrate it into a iOS megazord build.

Release is coming: https://github.com/mozilla/application-services/pull/3568
But currently on hold to land one more thing from the a-s team.
I'll keep this open to file follow-ups next week.

The new release including Glean should be building now and available shortly:

Flags: needinfo?(lina)
Blocks: 1664757
Blocks: 1664758
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: