Open Bug 1858740 Opened 7 months ago Updated 5 months ago

Look into adding an apple silicon workers and using it for building custom-car

Categories

(Testing :: Performance, task, P3)

Product:
Testing
Component:
Performance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: kshampur, Assigned: kshampur)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fxp] [relops-mac])

while spec wise the m2 workers are not as good as the linux/win workers (16gb ram, 4-8 cores i i recalll?), the silicon chip should prove to be a decent speed boost for building CaR osx (intel and mac)

we might be able to make an alias in here https://searchfox.org/mozilla-central/rev/e9b338c2d597067f99e96d5f20769f41f312fa8f/taskcluster/ci/config.yml#621

some more info here https://chromium.googlesource.com/chromium/src.git/+/main/docs/mac_arm64.md

similar to how intel mac can compile to arm64, we should be able to compile from arm64 to x64. (but if not, it should at least hopefully speed up arm64 mac builds, since they take about 11-12 hours on our intel machines)

Assignee: nobody → kshampur
Status: NEW → ASSIGNED

:aerickson/:masterwayz (not sure who is best to ask for this, please re direct as needed thanks)

What would be the best way to make a m2 worker available to do builds, similar to the b-osx-1015 workers? Am I on the right track to make that change in this file https://searchfox.org/mozilla-central/rev/e9b338c2d597067f99e96d5f20769f41f312fa8f/taskcluster/ci/config.yml#621?
Does anything need to be changed on https://github.com/mozilla-platform-ops/ronin_puppet?

some context: currently (when passing) these mac x64 CaR builds take about 7-8 hours using the b-osx-1015 worker. In Bug 1846849 we want to also build for Mac Arm64, and I've seen that take about ~11 hours on the intel mac minis

I am curious in how much of a speed gain we'd have if we used the m2 mac minis instead to do the builds (is it possible without too much work?)

Flags: needinfo?(mgoossens)
Flags: needinfo?(aerickson)
Flags: needinfo?(aerickson)
Whiteboard: [fxp] → [fxp] [relops-mac]

I saw this error in the logs - features: Additional property chainOfTrust is not allowed.

I think the issue is that you're normally running on a level 3 builder with chain of trust secrets and these workers are just testers (level 1 - no signing secrets). If this won't be released as a product, I think you can build on L1 workers (if so, it would just be a tweak to the taskgraph for this job to remove the chainOfTrust property in the payload).

We usually have dedicated pools for building, so if this is going to be frequent we should get a dedicated pool and workers.

Flags: needinfo?(mgoossens)

thanks :aerickson! I added a temporary workaround for the chainoftrust thing and that seems to help

:masterwayz I am encountering this error in this Try

[task 2023-10-19T22:40:55.601Z] ________ running 'python3 src/tools/clang/scripts/update.py' in '/opt/worker/tasks/task_169775284558467/custom_car/chromium'
[task 2023-10-19T22:40:55.601Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.601Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.601Z] Retrying in 5 s ...
[task 2023-10-19T22:40:55.601Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.601Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.601Z] Retrying in 10 s ...
[task 2023-10-19T22:40:55.601Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.601Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.601Z] Retrying in 20 s ...
[task 2023-10-19T22:40:55.601Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.601Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.601Z] Failed to download prebuilt clang package clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz
[task 2023-10-19T22:40:55.601Z] Use build.py if you want to build locally.
[task 2023-10-19T22:40:55.601Z] Exiting.
[task 2023-10-19T22:40:55.602Z] Error: Command 'python3 src/tools/clang/scripts/update.py' returned non-zero exit status 1 in /opt/worker/tasks/task_169775284558467/custom_car/chromium
[task 2023-10-19T22:40:55.602Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.602Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.602Z] Retrying in 5 s ...
[task 2023-10-19T22:40:55.602Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.602Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.602Z] Retrying in 10 s ...
[task 2023-10-19T22:40:55.602Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.602Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.602Z] Retrying in 20 s ...
[task 2023-10-19T22:40:55.602Z] Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Mac/clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz 
[task 2023-10-19T22:40:55.602Z] <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
[task 2023-10-19T22:40:55.602Z] Failed to download prebuilt clang package clang-llvmorg-18-init-8676-g11d07d9e-2.tar.xz
[task 2023-10-19T22:40:55.602Z] Use build.py if you want to build locally.
[task 2023-10-19T22:40:55.602Z] Exiting.

would you happen to know if these SSL certificate errors are at all related to any permission/network configurations on the silicon workers?

Flags: needinfo?(mgoossens)

was able to curl that file from an m1/m2 staging node. do we know which node specifically this test failed on?

Ryan, sorry - what does node mean in this context?

Ryan's test verifies the host is reachable and the SSL certs for that host are fine.

I think your issue is in your python environment. Is the certifi pip present in your virtualenv?

Thanks, adding certifi seems to have helped a bit!

I now have an issue with the xcode

[task 2023-10-24T16:40:21.397Z] xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
[task 2023-10-24T16:40:21.397Z] Traceback (most recent call last):
[task 2023-10-24T16:40:21.397Z]   File "/opt/worker/tasks/task_169816378248189/custom_car/chromium/src/build/config/apple/sdk_info.py", line 160, in <module>
[task 2023-10-24T16:40:21.397Z]     sys.exit(main())
[task 2023-10-24T16:40:21.397Z]   File "/opt/worker/tasks/task_169816378248189/custom_car/chromium/src/build/config/apple/sdk_info.py", line 145, in main
[task 2023-10-24T16:40:21.397Z]     FillXcodeVersion(settings, args.developer_dir)
[task 2023-10-24T16:40:21.397Z]   File "/opt/worker/tasks/task_169816378248189/custom_car/chromium/src/build/config/apple/sdk_info.py", line 62, in FillXcodeVersion
[task 2023-10-24T16:40:21.397Z]     lines = subprocess.check_output(['xcodebuild',
[task 2023-10-24T16:40:21.397Z]   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
[task 2023-10-24T16:40:21.397Z]     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
[task 2023-10-24T16:40:21.397Z]   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
[task 2023-10-24T16:40:21.397Z]     raise CalledProcessError(retcode, process.args,
[task 2023-10-24T16:40:21.397Z] subprocess.CalledProcessError: Command '['xcodebuild', '-version']' returned non-zero exit status 1.

in particular xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance not sure what to make of this - looking into it

we currently do not deploy the full xcode ide but have an open issue regarding this: https://mozilla-hub.atlassian.net/browse/RELOPS-680

Flags: needinfo?(mgoossens)

Ah that is great to know- thank you!

See Also: → 1868946
Blocks: 1869058
You need to log in before you can comment on or make changes to this bug.