Closed Bug 1711447 Opened 5 years ago Closed 5 years ago

Glean Swift: release it as a Swift Package

Categories

(Data Platform and Tools :: Glean: SDK, task, P2)

Product:
Data Platform and Tools
Component:
Glean: SDK
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: janerik, Assigned: janerik)

References

Details

Attachments

(3 files)

Link to GitHub pull-request: https://github.com/mozilla/glean/pull/1663
42 bytes, text/x-github-pull-request
Details | Review
PR 1668: Fix remaining issues for release
42 bytes, text/x-github-pull-request
Details | Review
PR 1669: iOS: Fix all occurences of Glean. in a line
42 bytes, text/x-github-pull-request
Details | Review

The iOS teams are exploring moving away from Carthage, and over to the Swift Package Manager.
One reason is that development on the new M1-based MacBooks is currently only possible using some workarounds really (or not at all in some cases).

If we switch over to SPM and provide a binary build of Glean as an XCFramework this should be easily consumable by those applications.

Background

The Swift Package Manager (SPM) is Apple's own tool for managing Swift code.
It brings most features you expect from a package manager, allows specifiying and resolving dependencies.
A single project can produce several targets/products.
It has direct integration into Xcode and therefore iOS development.

XCFrameworks are Apple's way of bundling multiple platform-specific variants of a project into a single package.
This allows to distribute the final build of a project without requiring the final application to know how to build that library.

The Swift Package Manager has the functionatility to depend on pre-built XCFrameworks.
See Distributing Binary Frameworks as Swift Packages for some guidance on that.

Complications

A binary target is a target of a product.
It's specified by name, URL and corresponding checksum (the SHA256 checksum of a Zip file).
A Swift package itself is a version tag in a Git repository.
Consumers specify this Git repository and a version constraint and the package manager resolves this at build time.

This makes it a catch-22:
We need the final XCFramework Zip to be able to generate its checksum.
But we need the checksum in the Package.swift, which is then tagged with the version in the Git repository.
So we can either put tag creation fully into the hands of CI and make that push the necessary changes.
Or pull out the wrapping Glean package into its own repository.

Solution

We keep another repository: glean-swift.
This contains nothing but the required Package.swift, pointing to the XCFramework release of Glean in the main repository.

  • Changes to the Glean repository
    • For now only the script to build an XCFramework, this can be done automatically at release time
    • The XCFramework can evenutall replace the bare framework zip
  • Test repository
    • The mentioned minimal repository with nothng but the Package.swift
    • A script updates the checksum and tags a release.
    • This can eventually be invoked automatically after a Glean release
  • Test release
    • xcframework manually uploaded for now
    • Consumers will reference this in their dependencies: 
      dependencies: [
    .package(url: "https://github.com/mozilla/glean-swift.git", from: "38.0.0"),
]

Sidenote

Rust currently only has Tier 3 support for the iOS simulator target on aarch64 platforms (M1 MacBooks).
It's therefore only available on Nightly with a self-built libstd.
I'm working on getting it to Tier 2 and thus to release eventually.

This means we need to build that target on Nightly.
But only that, others are built on stable.
As this is for testing only I don't think this is blocking.

Assignee: nobody → jrediger
Whiteboard: [telemetry:glean-rs:m?]

Thank you for investigating this!

So far we've taken the approach of fully binary frameworks. The current framework that Carthage pulls in just has a shared library and (generated) header files. The shared library is a mix of the compiled Rust code and the compiled Swift wrapper.

Because the shared library is a mix of Rust and Swift, the Xcode toolchain that it was built with must match the toolchain of the app. This is why we currently need to keep those in sync.

With Swift Packages there is an interesting option. Swift Packages are source-only packages. Xcode will pull in the package and then compile those sources as part of the app build. THis is great for a couple of reasons, but the best one is that it removes the compiler incompatibilities. You will always use the final Swift compiler. (And like you mention, Xcode is in charge to build for the right target, so all that complexity moves out of Glean builds)

What about the Rust code? I think that because Rust is basically at the "C Level", it is fine to include the Swift Wrapper in source but the glean.a for multiple architectures in binary form. Because glean.a is not created with the Swift compiler, it should be ABI compatible. Definitely between compiler upgrades like 12.4 to 12.5 (Swift 5.x to Swift 5.y)

But also maybe even between iOS and MacOS. I think it may be possible to create a glean.a for ARM64 that simply uses libc and works on both MacOS and iOS. If not, then I think there is a possibility to structure the binary part of an XCFramework in such a way that you don't just group libraries by their architecture (ARM64, x86_64) but also by their target platform (macOS, iOS, watchOS).

In any case, I think the most important part of using a Swift Package is to make sure it includes the Swift sources - that will remove the compiler incompatbilities and will radically simplify things. (It will also help a lot with debugging since now sources will be available and Xcode will generate symbols)

(In reply to Stefan Arentz | :st3fan | ⏰ EST | he/him from comment #2)

Thank you for investigating this!

So far we've taken the approach of fully binary frameworks. The current framework that Carthage pulls in just has a shared library and (generated) header files. The shared library is a mix of the compiled Rust code and the compiled Swift wrapper.

Because the shared library is a mix of Rust and Swift, the Xcode toolchain that it was built with must match the toolchain of the app. This is why we currently need to keep those in sync.

With Swift Packages there is an interesting option. Swift Packages are source-only packages. Xcode will pull in the package and then compile those sources as part of the app build. THis is great for a couple of reasons, but the best one is that it removes the compiler incompatibilities. You will always use the final Swift compiler. (And like you mention, Xcode is in charge to build for the right target, so all that complexity moves out of Glean builds)

What about the Rust code? I think that because Rust is basically at the "C Level", it is fine to include the Swift Wrapper in source but the glean.a for multiple architectures in binary form. Because glean.a is not created with the Swift compiler, it should be ABI compatible. Definitely between compiler upgrades like 12.4 to 12.5 (Swift 5.x to Swift 5.y)

I am all for that, but that will probably require more work on our side to restructure the project slightly.
That's why I went with the fully-binary release for now as a stepping stone there.
Long term I can definitely see us move to a mixed package, just the annoying way of how a binaryTarget in Package.swift is declared might makes this a bit annoying for us.

But also maybe even between iOS and MacOS. I think it may be possible to create a glean.a for ARM64 that simply uses libc and works on both MacOS and iOS. If not, then I think there is a possibility to structure the binary part of an XCFramework in such a way that you don't just group libraries by their architecture (ARM64, x86_64) but also by their target platform (macOS, iOS, watchOS).

Unlikely that the same arm64 build works on both iOS and macOS.
The Rust libstd differentiates between these two OSs, and additionally the compiler might also do different stuff based on the target.
But shipping an XCFramework for multiple platforms and architectures is no issue anymore.
The above linked XCFramework already ships for arm64-ios, arm64-ios-simulator and x86_64-ios (the Intel macbook simulator).

In any case, I think the most important part of using a Swift Package is to make sure it includes the Swift sources - that will remove the compiler incompatbilities and will radically simplify things. (It will also help a lot with debugging since now sources will be available and Xcode will generate symbols)

As said, agreed. Please do try to see how far the fully-binary approach gets us right now though. If that gets to be able to switch over Focus to SPM now, we have a bit more leeway for the rest of the refactoring

Thank you so much for putting time in this. For Focus iOS I will start with the glean-swift package and report back here how that is working out.

I've left my findings at https://github.com/mozilla-mobile/focus-ios/pull/1828#issuecomment-844379722

If it is better to discuss here, let me know.

Ok, here's a summary of the initial work and followups:

The plan would be:

This work can happen now, rather soon probably. This bug will track it.

Longerterm plan

  • Restructure the Swift part in https://github.com/mozilla/glean, add a Package.swift, which has the Swift code and the Rust part as targets.
  • Restructure our release process
    • On release pushes generate all artifacts (or only the xcframework artifact)
    • Modify Package.swift with the updated checksum
    • Create a new release (a git tag & github release) and attach the artifacts

This makes CI more complex, but the manual work is easier.
It's also required because the Package.swift needs the checksum and path to the xcframework artifact.

Type: defect → task
Priority: P3 → P2

Repository was moved to https://github.com/mozilla/glean-swift, pending bug 1713079 for admin access now.

Depends on: 1713079

Release done! (with a small hiccup, that should be gone for the next release)

https://github.com/mozilla/glean-swift/releases/tag/39.0.1

Waiting for confirmation that this works

v39.0.2 is out and it builds locally for me.
here's a focus PR: https://github.com/mozilla-mobile/focus-ios/pull/1907 (though bitrise failed with permission errors)

Blocks: 1714964
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: