1820216 - Build chromium official builds for MacOS

Status: RESOLVED FIXED

(Testing :: Raptor, task, P1)

Description

[Kash Shampur [:kshampur] ⌚EST]

Main things we'll need

• get a local build going on my M1
• configure xcode stuff if needed
• A script for toolchain task to build chromium in CI
• Add necessary build flags (to make it release + official build + PGO)
• Daily cron task of building
• Hook up new binary to raptor-browsertime tests (sp3)

Comment 1

[Kash Shampur [:kshampur] ⌚EST]

gets halfway through the build but looks like a cert issue is hit https://treeherder.mozilla.org/logviewer?job_id=408775847&repo=try&lineNumber=25748
I think there is a possible clang and/or macsdk version mismatch

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

dropping to p2 as I focus on linux and windows
mac is a bit slower to debug due to the frequent-ish sparse checkout failures in CI so I'll leave it for last

Comment 3

[Kash Shampur [:kshampur] ⌚EST]

It may make sense to hold off on this until Apple Silicon is set up with performance testing rather than focusing efforts on getting this running on the intel mac's

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

After discussing with sparky it is still important to have this on the intel Macs. Also we'd only have 4-5 silicon machines once those are running

Now that windows and mac are landed, I will resume this probably after the work week

Comment 5

[Kash Shampur [:kshampur] ⌚EST]

there is a potential possibility we can't build this on the 10.15 mac workers:
https://chromium.googlesource.com/chromium/src/+/main/docs/mac_build_instructions.md#system-requirements

in particular:

to check whether you have it, and what version you have. mac_sdk_official_version in mac_sdk.gni is the SDK version used on all the bots and for official builds, so that version is guaranteed to work. Building with a newer SDK usually works too (please fix or file a bug if it doesn't).

Building with an older SDK might also work, but if it doesn‘t then we won’t accept changes for making it work.

The easiest way to get the newest SDK is to use the newest version of Xcode, which often requires using the newest version of macOS. We don‘t use Xcode itself much, so if you’re know what you're doing, you can likely get the build working with an older version of macOS as long as you get a new version of the macOS SDK on it

So using a newer sdk on an older mac is not guaranteed to work, and could be a bit hacky. I am finding that the older sdk is not working

the 10.15 mac workers have macos 11 sdk, not 13. We do not have permissions to move the 13 sdk (from the existing toolchain fetches we have) into the right file path for what the chromium build scripts expect, /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs

I will look into if there is a possibility of getting this permission during run time

the current failure in using the 11 sdk:

[task 2023-05-19T05:23:22.837Z] FAILED: obj/net/net/x509_util_apple.o 
[task 2023-05-19T05:23:22.837Z] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/net/net/x509_util_apple.o.d -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -DCR_XCODE_VERSION=1220 -DCR_CLANG_REVISION=\"llvmorg-17-init-10134-g3da83fba-1\" -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -DCR_LIBCXX_REVISION=f8279b01085b800724f5c5629dc365b9f040dc53 -D_LIBCPP_ENABLE_ASSERTIONS=1 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DGOOGLE_PROTOBUF_INTERNAL_DONATE_STEAL_INLINE=0 -DHAVE_PTHREAD -DNET_IMPLEMENTATION -DENABLE_BUILT_IN_DNS -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -I../.. -Igen -I../../buildtools/third_party/libc++ -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../net/third_party/quiche/overrides -I../../net/third_party/quiche/src/quiche/common/platform/default -I../../net/third_party/quiche/src -Igen/net/third_party/quiche/src -I../../third_party/zlib -I../../third_party/ced/src -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/brotli/include -Wall -Werror -Wextra -Wimplicit-fallthrough -Wextra-semi -Wunreachable-code-aggressive -Wthread-safety -Wunguarded-availability -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wenum-compare-conditional -Wno-ignored-pragma-optimize -Wno-deprecated-builtins -Wno-bitfield-constant-conversion -Wno-deprecated-this-capture -Wshadow -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing -fstack-protector -femit-dwarf-unwind=no-compact-unwind -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -ffp-contract=off -fcomplete-member-pointers -arch x86_64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern -fno-omit-frame-pointer -g0 -isysroot ../../../../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -mmacos-version-min=10.13 -fvisibility=hidden -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ref-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang check-stack-allocated -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -O2 -DPROTOBUF_ALLOW_DEPRECATED=1 -Wexit-time-destructors -std=c++20 -Wno-trigraphs -fno-exceptions -fno-rtti -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include -fvisibility-inlines-hidden -include obj/net/net/precompile.h-cc -c ../../net/cert/x509_util_apple.cc -o obj/net/net/x509_util_apple.o
[task 2023-05-19T05:23:22.837Z] ../../net/cert/x509_util_apple.cc:148:9: error: use of undeclared identifier 'SecTrustCopyCertificateChain'; did you mean 'SecTrustGetCertificateCount'?
[task 2023-05-19T05:23:22.837Z]         SecTrustCopyCertificateChain(trust));
[task 2023-05-19T05:23:22.837Z]         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
[task 2023-05-19T05:23:22.837Z]         SecTrustGetCertificateCount
[task 2023-05-19T05:23:22.837Z] ../../../../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/System/Library/Frameworks/Security.framework/Headers/SecTrust.h:494:9: note: 'SecTrustGetCertificateCount' declared here
[task 2023-05-19T05:23:22.837Z] CFIndex SecTrustGetCertificateCount(SecTrustRef trust)
[task 2023-05-19T05:23:22.837Z]         ^
[task 2023-05-19T05:23:22.837Z] ../../net/cert/x509_util_apple.cc:147:12: error: no matching conversion for functional-style cast from 'CFIndex' (aka 'long') to 'base::ScopedCFTypeRef<CFArrayRef>' (aka 'ScopedTypeRef<const __CFArray *, internal::ScopedCFTypeRefTraits<const __CFArray *>>')
[task 2023-05-19T05:23:22.837Z]     return base::ScopedCFTypeRef<CFArrayRef>(
[task 2023-05-19T05:23:22.837Z]            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[task 2023-05-19T05:23:22.837Z] ../../base/mac/scoped_typeref.h:55:22: note: candidate constructor not viable: no known conversion from 'CFIndex' (aka 'long') to 'element_type' (aka 'const __CFArray *') for 1st argument
[task 2023-05-19T05:23:22.837Z]   explicit constexpr ScopedTypeRef(
[task 2023-05-19T05:23:22.837Z]                      ^
[task 2023-05-19T05:23:22.837Z] ../../base/mac/scoped_typeref.h:63:3: note: candidate constructor not viable: no known conversion from 'CFIndex' (aka 'long') to 'const ScopedTypeRef<const __CFArray *, ScopedCFTypeRefTraits<const __CFArray *>>' for 1st argument
[task 2023-05-19T05:23:22.837Z]   ScopedTypeRef(const ScopedTypeRef<T, Traits>& that)
[task 2023-05-19T05:23:22.837Z]   ^
[task 2023-05-19T05:23:22.837Z] ../../base/mac/scoped_typeref.h:77:3: note: candidate constructor not viable: no known conversion from 'CFIndex' (aka 'long') to 'ScopedTypeRef<const __CFArray *, ScopedCFTypeRefTraits<const __CFArray *>>' for 1st argument
[task 2023-05-19T05:23:22.837Z]   ScopedTypeRef(ScopedTypeRef<T, Traits>&& that) : object_(that.object_) {
[task 2023-05-19T05:23:22.837Z]   ^
[task 2023-05-19T05:23:22.837Z] ../../base/mac/scoped_typeref.h:71:12: note: candidate template ignored: could not match 'ScopedTypeRef<R, RTraits>' against 'CFIndex' (aka 'long')
[task 2023-05-19T05:23:22.837Z]   explicit ScopedTypeRef(const ScopedTypeRef<R, RTraits>& that_as_subclass)

and if one looks into the source code, there was some changes regarding SecTrustCopyCertificateChain and SecTrustGetCertificateCount' between 11 and 13, so it could be possible that using the 13 sdk on an older OS/xcode could be the solution (tbd)

Comment 6

[Kash Shampur [:kshampur] ⌚EST]

another update:

looks like SDK was indeed the issue. digging through the code I found you could specify a mac_sdk_path for gn so this let's me get past the previous issue I had of setting the sdk version. I tried sdk 10,11,13 (which we have toolchain fetches of) but all did not work either.

Recently 13.3 was landed in Bug 1833995, and using that 13.3 sdk it looks like it built (non PGO)

some issues:

• takes a long time (almost 7 hours) compared to windows and linux
• while the build itself completes (you can see the Tar artifact) the task itself fails.
• there is some auth/insufficientscope(?) issue where I can't click the tar file to download it
• depending on my tool-chain artifact path (public/build/<..> vs project/gecko/<...>) the error is different when clicking the tar? e.g. 1 and 2 from above
• occasionally there is an intermittent failure ModuleNotFoundError: No module named 'importlib_metadata' that occurs, e.g.

I will look into why the build completes but the task fails (initial guess is auth expires because task takes way too long?)

Comment 7

[Kash Shampur [:kshampur] ⌚EST]

https://treeherder.mozilla.org/jobs?repo=try&revision=9076d8fd7d8f408a710b61cec418b7e5700b4a8f&selectedTaskRun=S_F_C14KRdqMeuOsJ8Lt8Q.0

here the task is successfully passing. I have no idea why, but only zipping the /Chromium.app contents rather than everything (like in win & linux) seems to be the way to go?

the weird thing is the tar.zst file seems to be invalid (unzstd does not work)
renaming it as just tar, it seems to unzip fine.

Comment 8

[Kash Shampur [:kshampur] ⌚EST]

(guess) probably the -a flag for tar on the osx worker doesn't properly autodetect/recognize zst

Comment 9

[Kash Shampur [:kshampur] ⌚EST]

there seems to be a chromedriver issue, maybe mac permissions related?
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=292cd47035b4b81a887cea6a4b9aa592e1444b8b&selectedTaskRun=LLFGzICUSue8SZCN_lSy2Q.0

also .gz file for the CaR binary seems to be the way to go (.zst seems to be invalid on the worker that builds CaR)

Comment 10

[Kash Shampur [:kshampur] ⌚EST]

Also, w.r.t. Mac taking 6-7 hours (non pgo, and no symbol builds) it makes sense when compared to the win/linux workers. I found this information

• Mac has 16gb ram and 6 cores
• Linux has ~74GB RAM and 16 cores
• Windows has ~67 gb RAM 32 cores

So the build time discrepancy makes sense now. We did suspect the machine specs would be the reason, and seems to be the case.

Comment 11

[Kash Shampur [:kshampur] ⌚EST]

(In reply to Kash Shampur [:kshampur] ⌚EST from comment #9)

there seems to be a chromedriver issue, maybe mac permissions related?
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=292cd47035b4b81a887cea6a4b9aa592e1444b8b&selectedTaskRun=LLFGzICUSue8SZCN_lSy2Q.0

also .gz file for the CaR binary seems to be the way to go (.zst seems to be invalid on the worker that builds CaR)

Now I am seeing the speedometer3 tests pass!
https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=W-iWJg63Qn6z2iSqcMwoVg.0&revision=ebd2d716cd6fb1e3e18c0033287c235c351cedb4

only thing that really changed was gz -> xz compression, so I guess it was not a chromedriver or mac permission issue.
as for why this fixes it, no idea.

Comment 12

[Kash Shampur [:kshampur] ⌚EST]

and, PGO builds
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=b214b49865806859b662c4eaeed1cce0de3e5735&selectedTaskRun=MC6PGcIvTGe1v0wN13mlRA.0

Comment 13

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1820216 - Add toolchain task for Chromium-as-Release for OSX. r?#perftest

This patch provides a shell script to build a custom Chromium-as-Release (CaR) on OSX.
Additionally, we can remove the previous scripts that were used for win64 & linux64 and
refactor everything into a single script due to the similarities, and add OS platform specific logic

Comment 14

[Kash Shampur [:kshampur] ⌚EST]

Attachment: Bug 1820216 - Run Custom chromium-as-release performance tests on OSX. r?#perftest

Depends on D180824

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3c090815f3d2
Add toolchain task for Chromium-as-Release for OSX. r=perftest-reviewers,glandium,sparky
https://hg.mozilla.org/integration/autoland/rev/1f03620644bd
Run Custom chromium-as-release performance tests on OSX. r=perftest-reviewers,sparky

Comment 16

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/3c090815f3d2
https://hg.mozilla.org/mozilla-central/rev/1f03620644bd