1727559 - Intermittent Bp-hybrid RuntimeError: Error running dsymutil: LLVM ERROR: out of memory

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ctuns [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=349529302&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OrneiUWITNKYK4R4oMTN4g/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -  Running Mac pre-processing on file: /builds/worker/workspace/obj-build/toolkit/library/gtest/XUL
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -  /builds/worker/checkouts/gecko/build/macosx/llvm-dsymutil --arch=x86_64 /builds/worker/workspace/obj-build/toolkit/library/gtest/XUL
[task 2021-08-25T13:52:11.163Z] 13:52:11    ERROR -  Traceback (most recent call last):
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 1103, in <module>
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -      main()
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 1098, in main
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -      dumper.Process(args[2], options.count_ctors)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 533, in Process
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -      self.ProcessFile(file_to_process, count_ctors=count_ctors)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 892, in ProcessFile
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -      dsymbundle = self.GenerateDSYM(file)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py", line 936, in GenerateDSYM
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -      raise RuntimeError("Error running dsymutil: %s" % dsymerr)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -  RuntimeError: Error running dsymutil: LLVM ERROR: out of memory
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -  PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -   #0 0x0000000000f3c1da llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xf3c1da)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -   #1 0x0000000000f3a2b4 llvm::sys::RunSignalHandlers() (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xf3a2b4)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -   #2 0x0000000000f3a3f3 SignalHandler(int) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xf3a3f3)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -   #3 0x00007f11e2971140 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x14140)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -   #4 0x00007f11e2478ce1 raise (/lib/x86_64-linux-gnu/libc.so.6+0x3bce1)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -   #5 0x00007f11e2462537 abort (/lib/x86_64-linux-gnu/libc.so.6+0x25537)
[task 2021-08-25T13:52:11.163Z] 13:52:11     INFO -   #6 0x0000000000edcbe1 llvm::report_bad_alloc_error(char const*, bool) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xedcbe1)
[task 2021-08-25T13:52:11.164Z] 13:52:11     INFO -   #7 0x0000000000eecc28 llvm::SmallVectorBase<unsigned long>::grow_pod(void*, unsigned long, unsigned long) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xeecc28)
[task 2021-08-25T13:52:11.164Z] 13:52:11     INFO -   #8 0x0000000000a29a1f llvm::MCObjectStreamer::emitBytes(llvm::StringRef) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xa29a1f)
[task 2021-08-25T13:52:11.164Z] 13:52:11     INFO -   #9 0x0000000000a33557 llvm::MCStreamer::emitIntValue(unsigned long, unsigned int) (.localalias.288) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xa33557)
[task 2021-08-25T13:52:11.164Z] 13:52:11     INFO -  #10 0x00000000008db284 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db284)
[task 2021-08-25T13:52:11.164Z] 13:52:11     INFO -  #11 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.164Z] 13:52:11     INFO -  #12 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.165Z] 13:52:11     INFO -  #13 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.165Z] 13:52:11     INFO -  #14 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.165Z] 13:52:11     INFO -  #15 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.165Z] 13:52:11     INFO -  #16 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.165Z] 13:52:11     INFO -  #17 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.165Z] 13:52:11     INFO -  #18 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.165Z] 13:52:11     INFO -  #19 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.166Z] 13:52:11     INFO -  #20 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.166Z] 13:52:11     INFO -  #21 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.166Z] 13:52:11     INFO -  #22 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.166Z] 13:52:11     INFO -  #23 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.166Z] 13:52:11     INFO -  #24 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.166Z] 13:52:11     INFO -  #25 0x00000000008db2c2 llvm::AsmPrinter::emitDwarfDIE(llvm::DIE const&) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x8db2c2)
[task 2021-08-25T13:52:11.166Z] 13:52:11     INFO -  #26 0x00000000009d67cf llvm::DwarfStreamer::emitDIE(llvm::DIE&) (.localalias.274) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x9d67cf)
[task 2021-08-25T13:52:11.167Z] 13:52:11     INFO -  #27 0x00000000009d22a1 llvm::DWARFLinker::DIECloner::cloneAllCompileUnits(llvm::DWARFContext&, llvm::DwarfFile const&, llvm::StrongType<llvm::NonRelocatableStringpool, llvm::OffsetsTag>&, bool) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x9d22a1)
[task 2021-08-25T13:52:11.167Z] 13:52:11     INFO -  #28 0x00000000009d4b43 llvm::DWARFLinker::link()::'lambda0'(unsigned long)::operator()(unsigned long) const (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x9d4b43)
[task 2021-08-25T13:52:11.167Z] 13:52:11     INFO -  #29 0x00000000009d628c std::_Function_handler<void (), llvm::DWARFLinker::link()::'lambda3'()>::_M_invoke(std::_Any_data const&) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0x9d628c)
[task 2021-08-25T13:52:11.167Z] 13:52:11     INFO -  #30 0x0000000000ef7b81 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::function<void ()>, std::allocator<int>, void ()>::_M_run()::'lambda'(), void> >::_M_invoke(std::_Any_data const&) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xef7b81)
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -  #31 0x0000000000ef81cb std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xef81cb)
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -  #32 0x00007f11e296e34f __pthread_once_slow (/lib/x86_64-linux-gnu/libpthread.so.0+0x1134f)
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -  #33 0x0000000000ef8694 std::thread::_State_impl<std::thread::_Invoker<std::tuple<llvm::ThreadPool::ThreadPool(llvm::ThreadPoolStrategy)::'lambda'()> > >::_M_run() (/builds/worker/fetches/llvm-dsymutil/bin/dsymutil+0xef8694)
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -  #34 0x00007f11e2830ed0 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xceed0)
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -  #35 0x00007f11e2965ea7 start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8ea7)
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -  #36 0x00007f11e253adef clone (/lib/x86_64-linux-gnu/libc.so.6+0xfddef)
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -  Aborted
[task 2021-08-25T13:52:11.168Z] 13:52:11    ERROR -  Traceback (most recent call last):
[task 2021-08-25T13:52:11.168Z] 13:52:11     INFO -    File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -      return _run_code(code, main_globals, None,
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -    File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -      exec(code, run_globals)
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/dumpsymbols.py", line 107, in <module>
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -      sys.exit(main(sys.argv[1:]))
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/dumpsymbols.py", line 103, in main
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -      return dump_symbols(args.library_or_program, args.tracking_file, args.count_ctors)
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/dumpsymbols.py", line 83, in dump_symbols
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -      out_files = subprocess.check_output(args, universal_newlines=True)
[task 2021-08-25T13:52:11.169Z] 13:52:11     INFO -    File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
[task 2021-08-25T13:52:11.170Z] 13:52:11     INFO -      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
[task 2021-08-25T13:52:11.170Z] 13:52:11     INFO -    File "/usr/lib/python3.9/subprocess.py", line 528, in run
[task 2021-08-25T13:52:11.170Z] 13:52:11     INFO -      raise CalledProcessError(retcode, process.args,
[task 2021-08-25T13:52:11.170Z] 13:52:11     INFO -  subprocess.CalledProcessError: Command '['/builds/worker/workspace/obj-build/_virtualenvs/common/bin/python', '/builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py', '-c', '-a', 'x86_64', '--vcs-info', '--install-manifest=/builds/worker/workspace/obj-build/_build_manifests/install/dist_include,/builds/worker/workspace/obj-build/dist/include', '-s', '/builds/worker/checkouts/gecko', '/builds/worker/fetches/dump_syms/dump_syms', '/builds/worker/workspace/obj-build/dist/crashreporter-symbols', '/builds/worker/workspace/obj-build/toolkit/library/gtest/XUL']' returned non-zero exit status 1.
[task 2021-08-25T13:52:11.170Z] 13:52:11    ERROR -  gmake[4]: *** [/builds/worker/checkouts/gecko/config/rules.mk:662: XUL_syms.track] Error 1
[task 2021-08-25T13:52:11.170Z] 13:52:11     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/library/gtest'
[task 2021-08-25T13:52:11.170Z] 13:52:11    ERROR -  gmake[3]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:72: toolkit/library/gtest/syms] Error 2
[task 2021-08-25T13:52:11.171Z] 13:52:11     INFO -  gmake[3]: *** Waiting for unfinished jobs....
[task 2021-08-25T13:53:14.852Z] 13:53:14     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/toolkit/library/build'
[task 2021-08-25T13:53:14.853Z] 13:53:14     INFO -  /builds/worker/workspace/obj-build/_virtualenvs/common/bin/python -m mozbuild.action.dumpsymbols /builds/worker/workspace/obj-build/toolkit/library/build/XUL /builds/worker/workspace/obj-build/toolkit/library/build/XUL_syms.track --count-ctors
[task 2021-08-25T13:53:14.853Z] 13:53:14     INFO -  Running: /builds/worker/workspace/obj-build/_virtualenvs/common/bin/python /builds/worker/checkouts/gecko/toolkit/crashreporter/tools/symbolstore.py -c -a x86_64 --vcs-info --install-manifest=/builds/worker/workspace/obj-build/_build_manifests/install/dist_include,/builds/worker/workspace/obj-build/dist/include -s /builds/worker/checkouts/gecko /builds/worker/fetches/dump_syms/dump_syms /builds/worker/workspace/obj-build/dist/crashreporter-symbols /builds/worker/workspace/obj-build/toolkit/library/build/XUL --count-ctors
[task 2021-08-25T13:53:14.853Z] 13:53:14     INFO -  Starting Mac pre-processing on file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL
[task 2021-08-25T13:53:14.853Z] 13:53:14     INFO -  Running Mac pre-processing on file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL
[task 2021-08-25T13:53:14.853Z] 13:53:14     INFO -  /builds/worker/checkouts/gecko/build/macosx/llvm-dsymutil --arch=x86_64 /builds/worker/workspace/obj-build/toolkit/library/build/XUL
[task 2021-08-25T13:53:14.853Z] 13:53:14     INFO -  Finished processing /builds/worker/workspace/obj-build/toolkit/library/build/XUL in 243.90s
[task 2021-08-25T13:53:14.854Z] 13:53:14     INFO -  Beginning work for file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL
[task 2021-08-25T13:53:14.854Z] 13:53:14     INFO -  Processing file: /builds/worker/workspace/obj-build/toolkit/library/build/XUL

Comment 1

[Intermittent Failures Robot]

1 failures in 2580 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• osx-cross: 1
  • plain: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-08-23&endday=2021-08-29&tree=all

Comment 2

[Andi [:andi]]

How often does this happen?

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

3 times since the bug has been filed: The two classifications and this Try failure.

Comment 4

[Intermittent Failures Robot]

1 failures in 3084 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• osx-cross: 1
  • plain: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-08-30&endday=2021-09-05&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 3266 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• osx-cross: 1
  • plain: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-09-06&endday=2021-09-12&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 3375 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• osx-cross: 1
  • plain: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-09-13&endday=2021-09-19&tree=all

Comment 7

[Intermittent Failures Robot]

1 failures in 3376 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• osx-cross: 1
  • plain: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-09-27&endday=2021-10-03&tree=all

Comment 8

[Intermittent Failures Robot]

2 failures in 3626 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• osx-cross: 2
  • plain: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-10-04&endday=2021-10-10&tree=all

Comment 9

[Mike Hommey [:glandium]]

llvm-dsymutil is notably both slow and memory-intensive (which is absurd because it's not theoretically doing much more work than lld does in a linux build, and lld doesn't have this problem)
https://firefoxci.taskcluster-artifacts.net/PumOQWKFR9qUt6f3G-a_3A/2/public/build/build_resources.html

Comment 10

[Mike Hommey [:glandium]]

There was upstream discussion to make dsymutil use the lld machinery a long time ago...
https://lists.llvm.org/pipermail/llvm-dev/2015-November/092527.html

Comment 11

[Intermittent Failures Robot]

2 failures in 3315 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• osx-cross: 2
  • plain: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-10-11&endday=2021-10-17&tree=all

Comment 12

[Intermittent Failures Robot]

3 failures in 3186 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• osx-cross: 3
  • plain: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-10-18&endday=2021-10-24&tree=all

Comment 13

[Mike Hommey [:glandium]]

Attachment: Bug 1727559 - Disable building symbols on hybrid builds.

That will save time and avoid hitting memory limits on mac builds. We
don't need them.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0c9f7bd4c4b8
Disable building symbols on hybrid builds. r=firefox-build-system-reviewers,andi

Comment 15

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/0c9f7bd4c4b8

Comment 16

[Intermittent Failures Robot]

3 failures in 2845 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• osx-cross: 3
  • plain: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727559&startday=2021-10-25&endday=2021-10-31&tree=all