Closed Bug 1934191 Opened 6 days ago Closed 19 hours ago

"mach build" for debug artifact builds on MacOS ARM machines downloads the x86_64 build instead of aarch64

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(firefox135 fixed)

Status:
VERIFIED FIXED
Milestone:
135 Branch
Tracking Flags:
Tracking Status
firefox135 --- fixed

People

(Reporter: whimboo, Assigned: glandium)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Bug 1934191 - Handle aarch64-apple-darwin artifact builds.
48 bytes, text/x-phabricator-request
Details | Review

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:135.0) Gecko/20100101 Firefox/135.0 ID:20241128215411

I’ve observed consistent shutdown hangs in Firefox when running web-platform tests locally with a Firefox debug build. However, this issue doesn’t occur when I either avoid using the profiler environment variables or switch to an opt build.

Here is the command that I'm running:

MOZ_PROFILER_STARTUP=1 MOZ_PROFILER_SHUTDOWN=/Users/henrik/profiles/profile.json mach wpt --setpref="remote.log.truncate=false" --webdriver-arg=-vv --webdriver-binary=target/debug/geckodriver testing/web-platform/tests/webdriver/tests/classic/perform_actions/pointer_touch.py

Due to the shutdown hang and the final crash due to the shutdown hang monitor the profile never gets stored on disk. As such I tried to use the Instruments.app but recording a Time Profile shows no symbols in the stacks, which makes it useless.

Here at least the related log output:

 0:30.00 pid:22649 1732871071301	Marionette	DEBUG	0 <- [1,389,null,{"cause":"shutdown","forced":false,"in_app":true}]
 0:30.05 pid:22649 [Child 22664, Main Thread] WARNING: DispatchEvent called on non-current inner window, dropping. Please check the window in the caller instead.: file /builds/worker/checkouts/gecko/dom/base/nsGlobalWindowInner.cpp:4149
 0:30.05 pid:22649 [Child 22664, Main Thread] WARNING: DispatchEvent called on non-current inner window, dropping. Please check the window in the caller instead.: file /builds/worker/checkouts/gecko/dom/base/nsGlobalWindowInner.cpp:4149
 0:30.05 pid:22649 [Child 22664, Main Thread] WARNING: DispatchEvent called on non-current inner window, dropping. Please check the window in the caller instead.: file /builds/worker/checkouts/gecko/dom/base/nsGlobalWindowInner.cpp:4149
 0:30.08 pid:22649 1732871071380	webdriver::server	DEBUG	Teardown session
 0:30.11 pid:22649 1732871071404	Marionette	DEBUG	Closed connection 0
 0:30.12 pid:22649 [Parent 22653, Socket Thread] WARNING: cannot post event if not initialized: file /builds/worker/checkouts/gecko/netwerk/protocol/http/nsHttpConnectionMgr.cpp:240
 0:30.37 pid:22649 [Parent 22653, Main Thread] WARNING: BlockShutdown: Init failed: 'NS_FAILED(rv)', file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtectionStorage.cpp:366
 0:30.44 pid:22649 [Child 22664, Main Thread] WARNING: IPC message 'PContent::Msg_StoreUserInteractionAsPermission' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.44 pid:22649 [Child 22664, Main Thread] WARNING: NS_ENSURE_TRUE(aWindowContext) failed: file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtection.cpp:437
 0:30.44 pid:22649 [Child 22664, Main Thread] WARNING: IPC message 'PContent::Msg_StoreUserInteractionAsPermission' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: NS_ENSURE_TRUE(aWindowContext) failed: file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtection.cpp:437
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: IPC message 'PContent::Msg_StoreUserInteractionAsPermission' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: NS_ENSURE_TRUE(aWindowContext) failed: file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtection.cpp:437
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: IPC message 'PContent::Msg_StoreUserInteractionAsPermission' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: NS_ENSURE_TRUE(aWindowContext) failed: file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtection.cpp:437
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: IPC message 'PContent::Msg_StoreUserInteractionAsPermission' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: NS_ENSURE_TRUE(aWindowContext) failed: file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtection.cpp:437
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: IPC message 'PContent::Msg_StoreUserInteractionAsPermission' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: NS_ENSURE_TRUE(aWindowContext) failed: file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtection.cpp:437
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: IPC message 'PContent::Msg_StoreUserInteractionAsPermission' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.45 pid:22649 [Child 22664, Main Thread] WARNING: NS_ENSURE_TRUE(aWindowContext) failed: file /builds/worker/checkouts/gecko/toolkit/components/antitracking/bouncetrackingprotection/BounceTrackingProtection.cpp:437
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.000004s: startCrashBrowserTimer', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.000044s: initiateShutdownWorkThreads', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.000053s: starting', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: cache: '0.000076s: starting', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: sdb: '0.000109s: starting', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: fs: '0.000127s: starting', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: ls: '0.000137s: starting', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.000150s: startKillActorsTimer', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.000224s: Live database entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.000231s: gLiveDatabaseHashtable entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.000734s: Live database entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.000753s: gLiveDatabaseHashtable entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.000983s: Live database entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.000996s: gLiveDatabaseHashtable entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.001162s: Live database entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.001218s: Live database entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.001225s: gLiveDatabaseHashtable entry removed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, QuotaManager IO] WARNING: quota manager shutdown step: '0.001410s: ShutdownStorageOp::DoDirectoryWork -> ShutdownStorageInternal.', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.002341s: stopKillActorsTimer', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.002357s: shutdownAndJoinWorkThreads', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: idb: '0.002363s: completed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: cache: '0.002932s: completed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: sdb: '0.002947s: completed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: fs: '0.002953s: completed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: ls: '0.002958s: completed', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.002963s: shutdownAndJoinIOThread', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.003060s: invalidatePendingDirectoryLocks', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.45 pid:22649 [Parent 22653, IPDL Background] WARNING: quota manager shutdown step: '0.003069s: stopCrashBrowserTimer', file /builds/worker/checkouts/gecko/dom/quota/ActorsParent.cpp:2233
 0:30.65 pid:22649 [Utility 22659, Main Thread] WARNING: IPC message 'PUtilityProcess::Msg_FOGData' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.71 pid:22649 [Socket 22654, Main Thread] WARNING: IPC message 'PSocketProcess::Msg_FOGData' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:30.72 pid:22649 [RDD 22656, Main Thread] WARNING: IPC message 'PRDD::Msg_FOGData' discarded: actor cannot send: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:557
 0:31.82 pid:22649 [Parent 22653, Main Thread] WARNING: NS_ENSURE_SUCCESS_VOID(rv) failed with result 0x8000FFFF (NS_ERROR_UNEXPECTED): file /builds/worker/checkouts/gecko/dom/file/uri/BlobURLProtocolHandler.cpp:406
 0:31.82 pid:22649 [Parent 22653, Main Thread] WARNING: NS_ENSURE_TRUE(!!svc) failed: file /builds/worker/checkouts/gecko/dom/file/uri/BlobURLProtocolHandler.cpp:506
 1:36.93 pid:22649 XPCOM_MEM_BLOAT_LOG: /var/folders/sd/hgkqcwv90sz_q1pqjww78nfm0000gn/T/tmp3d1rfc08.mozrunner/runtests_leaks_22549.log
 1:36.93 pid:22649 Writing to log: /var/folders/sd/hgkqcwv90sz_q1pqjww78nfm0000gn/T/tmp3d1rfc08.mozrunner/runtests_leaks_22549.log
 1:36.94 pid:22649 RunWatchdog: Mainthread nested event loops during hang:
 1:36.94 pid:22649  --- (no nested event loop active)
 1:36.94 pid:22649 [22653] Hit MOZ_CRASH(Shutdown hanging at step CCPostLastCycleCollection. Something is blocking the main-thread.) at /builds/worker/checkouts/gecko/toolkit/components/terminator/nsTerminator.cpp:244
Initializing stack-fixing for the first stack frame, this may take a while...
 1:37.13 pid:22649 #01: mozilla::(anonymous namespace)::RunWatchdog(void*) (/Users/henrik/code/gecko-obj/debug/dist/NightlyDebug.app/Contents/MacOS/XUL + 0x6f5f4a3)
 1:37.13 pid:22649 #02: _pt_root (/Users/henrik/code/gecko-obj/debug/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib + 0x1b4c58)
 1:37.13 pid:22649 #03: _pthread_start (/usr/lib/system/libsystem_pthread.dylib + 0x618b)

When this happens I tried to at least to sample the hanging process via the activity monitor and that is the output:

Analysis of sampling plugin-container (pid 25824) every 1 millisecond
Process:         plugin-container [25824]
Path:            /Users/USER/*/NightlyDebug.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container
Load Address:    0x102907000
Identifier:      plugin-container
Version:         ???
Code Type:       X86-64 (translated)
Platform:        macOS
Parent Process:  firefox [25815]

Date/Time:       2024-11-29 10:51:22.486 +0100
Launch Time:     2024-11-29 10:50:54.873 +0100
OS Version:      macOS 14.7.1 (23H222)
Report Version:  7
Analysis Tool:   /usr/bin/sample

Physical footprint:         35.2M
Physical footprint (peak):  62.3M
Idle exit:                  untracked
----

Call graph:
    2429 Thread_292664   DispatchQueue_1: com.apple.main-thread  (serial)
    + 2429 start  (in dyld) + 1909  [0x2029a7345]
    +   2429 main  (in plugin-container) + 122  [0x10290770a]
    +     2429 XRE_InitChildProcess(int, char**, XREChildData const*)  (in XUL) + 4200  [0x122e5de58]
    +       2429 mozilla::ShutdownXPCOM(nsIServiceManager*)  (in XUL) + 1498  [0x11c681baa]
    +         2429 NSS_Shutdown  (in libnss3.dylib) + 104  [0x10b8a6538]
    +           2429 nss_Shutdown  (in libnss3.dylib) + 225  [0x10b8a6431]
    +             2429 SECMOD_Shutdown  (in libnss3.dylib) + 177  [0x10b8f7b31]
    +               2429 SECMOD_DestroyModule  (in libnss3.dylib) + 101  [0x10b8f7cb5]
    +                 2429 PK11_DestroySlot  (in libnss3.dylib) + 160  [0x10b8f11a0]
    +                   2429 SECMOD_SlotDestroyModule  (in libnss3.dylib) + 89  [0x10b8f8bd9]
    +                     2429 SECMOD_UnloadModule  (in libnss3.dylib) + 254  [0x10b8d58be]
    +                       2429 NSC_Finalize  (in libsoftokn3.dylib) + 20  [0x11311cf04]
    +                         2429 nsc_CommonFinalize  (in libsoftokn3.dylib) + 104  [0x11311cec8]
    +                           2429 BL_Unload  (in libsoftokn3.dylib) + 49  [0x11310b111]
    +                             2429 PR_UnloadLibrary  (in libnss3.dylib) + 58  [0x10b86358a]
    +                               2429 dyld4::APIs::dlclose(void*)  (in dyld) + 178  [0x2029d5f94]
    +                                 2429 dyld4::RuntimeState::decDlRefCount(dyld4::Loader const*)  (in dyld) + 217  [0x2029b0cd7]
    +                                   2429 dyld4::RuntimeState::garbageCollectInner()  (in dyld) + 476  [0x2029b143c]
    +                                     2429 dyld4::RuntimeState::removeLoaders(std::span<dyld4::Loader const*, 18446744073709551615ul> const&)  (in dyld) + 73  [0x2029b1bdf]
    +                                       2429 ???  (in Rosetta JIT)  [0x1029862c0]
    +                                         2429 ???  (in Rosetta Runtime Routines)  [0x102925070]
    +                                           2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x16900  [0x7ff7ffdea900]
    +                                             2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0xbb60  [0x7ff7ffddfb60]
    +                                               2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x13b14  [0x7ff7ffde7b14]
    +                                                 2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x42dc  [0x7ff7ffdd82dc]
    2429 Thread_292665: com.apple.rosetta.exceptionserver
    + 2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x4414  [0x7ff7ffdd8414]
    2429 Thread_292674: AsyncSignalControlThread
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 AsyncSignalControlThreadEntry(void*)  (in XUL) + 29  [0x1228e8fcd]
    +       2429 AsyncSignalControlThread::Watch()  (in XUL) + 71  [0x1228e9027]
    +         2429 read  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdcace]
    +           2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292676: SamplerThread
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 ThreadEntry(void*)  (in XUL) + 9  [0x1228bc439]
    +       2429 SamplerThread::Run()  (in XUL) + 4559  [0x1228b7bef]
    +         2429 thread_suspend  (in libsystem_kernel.dylib) + 84  [0x7ff80d00288c]
    +           2429 mach_msg2_trap  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdc06e]
    +             2429 ???  (in <unknown binary>)  [0x7ff89d2cec20]
    +               2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x15040  [0x7ff7ffde9040]
    +                 2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x14ef4  [0x7ff7ffde8ef4]
    +                   2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x162a8  [0x7ff7ffdea2a8]
    +                     2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x42dc  [0x7ff7ffdd82dc]
    2429 Thread_292683: Breakpad ExceptionHandler
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 google_breakpad::ExceptionHandler::WaitForMessage(void*)  (in XUL) + 207  [0x122e1996f]
    +       2429 mach_msg  (in libsystem_kernel.dylib) + 19  [0x7ff80cfdc35d]
    +         2429 mach_msg_overwrite  (in libsystem_kernel.dylib) + 653  [0x7ff80cfe3158]
    +           2429 mach_msg2_internal  (in libsystem_kernel.dylib) + 84  [0x7ff80cfea858]
    +             2429 mach_msg2_trap  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdc06e]
    +               2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292686: IPC I/O Child
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 ThreadFunc(void*)  (in XUL) + 10  [0x11d0a95aa]
    +       2429 base::Thread::ThreadMain()  (in XUL) + 687  [0x11d0ae6ff]
    +         2429 MessageLoop::Run()  (in XUL) + 99  [0x11d0a06c3]
    +           2429 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*)  (in XUL) + 696  [0x11d0a3df8]
    +             2429 event_base_loop  (in XUL) + 843  [0x11d0d89ab]
    +               2429 kq_dispatch  (in XUL) + 932  [0x11d0db034]
    +                 2429 kevent  (in libsystem_kernel.dylib) + 10  [0x7ff80cfe0d4e]
    +                   2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292687
    + 2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x42dc  [0x7ff7ffdd82dc]
    2429 Thread_292692
    + 2429 ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x42dc  [0x7ff7ffdd82dc]
    2429 Thread_292789: TaskController #0
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
    +       2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
    +         2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
    +           2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
    +             2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
    +               2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
    +                 2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292790: TaskController #1
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
    +       2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
    +         2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
    +           2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
    +             2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
    +               2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
    +                 2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292791: TaskController #2
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
    +       2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
    +         2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
    +           2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
    +             2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
    +               2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
    +                 2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292792: TaskController #3
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
    +       2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
    +         2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
    +           2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
    +             2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
    +               2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
    +                 2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292793: TaskController #4
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
    +       2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
    +         2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
    +           2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
    +             2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
    +               2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
    +                 2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292794: TaskController #5
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
    +       2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
    +         2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
    +           2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
    +             2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
    +               2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
    +                 2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292795: TaskController #6
    + 2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
    +   2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
    +     2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
    +       2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
    +         2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
    +           2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
    +             2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
    +               2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
    +                 2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]
    2429 Thread_292796: TaskController #7
      2429 thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
        2429 _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
          2429 _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
            2429 mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
              2429 mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
                2429 mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]
                  2429 _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
                    2429 __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
                      2429 ???  (in <unknown binary>)  [0x7ff89d2cea78]

Total number in stack (recursive counted multiple, when >=5):
        12       _pthread_start  (in libsystem_pthread.dylib) + 99  [0x7ff80d01c18b]
        12       thread_start  (in libsystem_pthread.dylib) + 15  [0x7ff80d017ae3]
        11       ???  (in <unknown binary>)  [0x7ff89d2cea78]
        8       __psynch_cvwait  (in libsystem_kernel.dylib) + 10  [0x7ff80cfdec1a]
        8       _pt_root  (in libnss3.dylib) + 264  [0x10b881c58]
        8       _pthread_cond_wait  (in libsystem_pthread.dylib) + 1211  [0x7ff80d01c6f3]
        8       mozilla::OffTheBooksCondVar::Wait(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>)  (in XUL) + 137  [0x11c607f09]
        8       mozilla::TaskController::RunPoolThread(mozilla::PoolThread*)  (in XUL) + 1345  [0x11c60f8b1]
        8       mozilla::detail::ConditionVariableImpl::wait_for(mozilla::detail::MutexImpl&, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator> const&)  (in libmozglue.dylib) + 77  [0x10b4ababd]

Sort by top of stack, same collapsed (when >= 5):
        ???  (in <unknown binary>)  [0x7ff89d2cea78]        26719
        ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x42dc  [0x7ff7ffdd82dc]        9716
        ???  (in runtime)  load address 0x7ff7ffdd4000 + 0x4414  [0x7ff7ffdd8414]        2429

Does it mean that we may hang somewhere in Rosetta?

Looks like it, yes. Why are you building for x86_64?

Hm, I do not build for x86_6 specifically. Here is what I have in my mozconfig:

mk_add_options MOZ_OBJDIR=/Users/henrik/code/gecko-obj/debug
mk_add_options AUTOCLOBBER=1

ac_add_options --enable-application=browser
ac_add_options --enable-artifact-builds
ac_add_options --enable-debug

Maybe the artifact build doesn't download the aarch64 but x86_64 binaries? That might explain why it also takes ages for me to actually get Firefox started for the first time. It's taking about 1min before the app is launching.

Glandium, do you know what the status of artifact builds on macOS aarch64 is?

Flags: needinfo?(mh+mozilla)

Oh indeed! I mounted the downloaded DMG and the file command outputs the following:

/Volumes/Firefox Nightly/Firefox NightlyDebug.app/Contents/MacOS/firefox: Mach-O 64-bit executable x86_64

Blocks: mach-busted
Component: Gecko Profiler → General
Product: Core → Firefox Build System
Summary: Constant shutdown hangs when running wpt tests and having the startup and shutdown profiler enabled → "mach build" for artifact builds on MacOS ARM machines downloads the x86_64 build instead of aarch64

And there for the final binary in the obj directory:

debug:

✗ file ../gecko-obj/debug/dist/bin/firefox 
../gecko-obj/debug/dist/bin/firefox: Mach-O 64-bit executable x86_64

opt:

✗ file ../gecko-obj/opt/dist/bin/firefox
../gecko-obj/opt/dist/bin/firefox: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
../gecko-obj/opt/dist/bin/firefox (for architecture x86_64):	Mach-O 64-bit executable x86_64
../gecko-obj/opt/dist/bin/firefox (for architecture arm64):	Mach-O 64-bit executable arm64
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Flags: needinfo?(mh+mozilla)
Summary: "mach build" for artifact builds on MacOS ARM machines downloads the x86_64 build instead of aarch64 → "mach build" for debug artifact builds on MacOS ARM machines downloads the x86_64 build instead of aarch64
Pushed by mh@glandium.org: https://hg.mozilla.org/integration/autoland/rev/0dd286ce921c Handle aarch64-apple-darwin artifact builds. r=firefox-build-system-reviewers,ahochheiden

https://hg.mozilla.org/mozilla-central/rev/0dd286ce921c

Status: ASSIGNED → RESOLVED
Closed: 19 hours ago
status-firefox135: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 135 Branch

This works great! Instead of around 1 minute for the very first start of a freshly build Firefox debug build it now takes only 3 seconds to have a Webdriver client connected. Thanks a lot.

Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: