1925510 - Intermittent GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)

Status: RESOLVED FIXED

(Core :: Gecko Profiler, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: tszentpeteri [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=478856886&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/adM3KLlORGK82YR6Kfxctg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-10-18T11:51:19.874Z] 11:51:19     INFO -  TEST-START | GeckoProfiler.AllThreads
[task 2024-10-18T11:51:20.037Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-18T11:51:20.038Z] 11:51:20     INFO -  Google Test trace:
[task 2024-10-18T11:51:20.038Z] 11:51:20     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:4682: Features: StackWalk Threads CPUAllThreads
[task 2024-10-18T11:51:20.038Z] 11:51:20     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-18T11:51:20.173Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-18T11:51:20.173Z] 11:51:20     INFO -  Google Test trace:
[task 2024-10-18T11:51:20.173Z] 11:51:20     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:4682: Features: StackWalk Threads SamplingAllThreads
[task 2024-10-18T11:51:20.174Z] 11:51:20     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-18T11:51:20.309Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-18T11:51:20.310Z] 11:51:20     INFO -  Google Test trace:
[task 2024-10-18T11:51:20.311Z] 11:51:20     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:4682: Features: StackWalk Threads CPUAllThreads SamplingAllThreads
[task 2024-10-18T11:51:20.311Z] 11:51:20     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-18T11:51:20.442Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-18T11:51:20.442Z] 11:51:20     INFO -  Google Test trace:
[task 2024-10-18T11:51:20.443Z] 11:51:20     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:4682: Features: StackWalk Threads MarkersAllThreads
[task 2024-10-18T11:51:20.443Z] 11:51:20     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-18T11:51:20.576Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-18T11:51:20.578Z] 11:51:20     INFO -  Google Test trace:
[task 2024-10-18T11:51:20.578Z] 11:51:20     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:4682: Features: StackWalk Threads CPUAllThreads MarkersAllThreads
[task 2024-10-18T11:51:20.578Z] 11:51:20     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-18T11:51:20.714Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-18T11:51:20.714Z] 11:51:20     INFO -  Google Test trace:
[task 2024-10-18T11:51:20.715Z] 11:51:20     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:4682: Features: StackWalk Threads SamplingAllThreads MarkersAllThreads
[task 2024-10-18T11:51:20.715Z] 11:51:20     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-18T11:51:20.850Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-18T11:51:20.850Z] 11:51:20     INFO -  Google Test trace:
[task 2024-10-18T11:51:20.851Z] 11:51:20     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:4682: Features: StackWalk Threads CPUAllThreads SamplingAllThreads MarkersAllThreads
[task 2024-10-18T11:51:20.852Z] 11:51:20     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-18T11:51:20.852Z] 11:51:20  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | test completed (time: 976ms)
[task 2024-10-18T11:51:20.853Z] 11:51:20     INFO -  TEST-START | GeckoProfiler.FailureHandling

Comment 1

[Intermittent Failures Robot]

1 failures in 4579 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• linux1804-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1925510&startday=2024-10-14&endday=2024-10-20&tree=all

Comment 2

[Intermittent Failures Robot]

3 failures in 4456 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform and build breakdown:

• linux1804-32-shippable-qr: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1925510&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 3

[Intermittent Failures Robot]

3 failures in 4456 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform and build breakdown:

• linux1804-32-shippable-qr: 1
  • opt: 1
• windows11-32-2009-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1925510&startday=2024-10-21&endday=2024-10-27&tree=all

Comment 4

[agoloman]

Tier 1 link failure here

Comment 5

[Sandor Molnar[:smolnar]]

Backfills and retriggers indicate it's coming from bug 1923595

@Olli, could you take a look into this? seems kinda frequent.
here's a failure log

TEST-START | GeckoProfiler.AllThreads
[task 2024-10-26T20:40:28.177Z] 20:40:28  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-26T20:40:28.179Z] 20:40:28     INFO -  Google Test trace:
[task 2024-10-26T20:40:28.179Z] 20:40:28     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp(4682): Features: StackWalk Threads CPUAllThreads
[task 2024-10-26T20:40:28.179Z] 20:40:28     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-26T20:40:28.861Z] 20:40:28  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-26T20:40:28.863Z] 20:40:28     INFO -  Google Test trace:
[task 2024-10-26T20:40:28.863Z] 20:40:28     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp(4682): Features: StackWalk Threads SamplingAllThreads
[task 2024-10-26T20:40:28.863Z] 20:40:28     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-26T20:40:29.570Z] 20:40:29  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-26T20:40:29.572Z] 20:40:29     INFO -  Google Test trace:
[task 2024-10-26T20:40:29.572Z] 20:40:29     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp(4682): Features: StackWalk Threads CPUAllThreads SamplingAllThreads
[task 2024-10-26T20:40:29.573Z] 20:40:29     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-26T20:40:30.277Z] 20:40:30  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-26T20:40:30.279Z] 20:40:30     INFO -  Google Test trace:
[task 2024-10-26T20:40:30.280Z] 20:40:30     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp(4682): Features: StackWalk Threads MarkersAllThreads
[task 2024-10-26T20:40:30.280Z] 20:40:30     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-26T20:40:30.972Z] 20:40:30  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-26T20:40:30.974Z] 20:40:30     INFO -  Google Test trace:
[task 2024-10-26T20:40:30.974Z] 20:40:30     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp(4682): Features: StackWalk Threads CPUAllThreads MarkersAllThreads
[task 2024-10-26T20:40:30.975Z] 20:40:30     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-26T20:40:31.663Z] 20:40:31  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-26T20:40:31.665Z] 20:40:31     INFO -  Google Test trace:
[task 2024-10-26T20:40:31.665Z] 20:40:31     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp(4682): Features: StackWalk Threads SamplingAllThreads MarkersAllThreads
[task 2024-10-26T20:40:31.665Z] 20:40:31     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-26T20:40:32.379Z] 20:40:32  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | Expected: (aOutput) != (nullptr), actual: NULL vs (nullptr)
[task 2024-10-26T20:40:32.381Z] 20:40:32     INFO -  Google Test trace:
[task 2024-10-26T20:40:32.381Z] 20:40:32     INFO -  /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp(4682): Features: StackWalk Threads CPUAllThreads SamplingAllThreads MarkersAllThreads
[task 2024-10-26T20:40:32.381Z] 20:40:32     INFO -   @ /builds/worker/checkouts/gecko/tools/profiler/tests/gtest/GeckoProfiler.cpp:1522
[task 2024-10-26T20:40:32.382Z] 20:40:32  WARNING -  TEST-UNEXPECTED-FAIL | GeckoProfiler.AllThreads | test completed (time: 4958ms)

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1923595

Comment 7

[Olli Pettay [:smaug][[e-mail]]]

I don't understand how the backout in bug 1923595 could have caused this.
And this is failing on windows11-32 only. Are windows11-32 tests something new?

Nazim, the test might be a bit more familiar to you. Any ideas here?

Comment 8

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Interesting. It doesn't look like it could be triggered by Bug 1923595. I pushed a try run to see the errors, maybe they could give us more insight: https://treeherder.mozilla.org/jobs?repo=try&revision=1d0de815f623ef0c3e44464fb9dd05a52e6975b4 I'll investigate more later.

Comment 9

[Intermittent Failures Robot]

63 failures in 4361 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #31 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 46
• mozilla-central: 16
• try: 1

Platform and build breakdown:

• windows11-32-2009-qr: 47
  • opt: 47
• windows11-32-2009-shippable-qr: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1925510&startday=2024-10-28&endday=2024-11-03&tree=all

Comment 10

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

This test log shows that the tests are failing due to OOM: https://treeherder.mozilla.org/logviewer?job_id=481027569&repo=try&lineNumber=39755

aWriter failed with: OOM in ChunkedJSONWriteFunc::AllocChunk allocating new chunk

It doesn't look like it's directly related to bug 1923595. But this bug seems to change how we calculate the number of cores, and probably now the result is more, and creating more threads. Since we have more threads, we are trying to serialize more data, and it seems like we reached this threshold on these platforms.

But considering that these failures are happening only on 32 bit windows and linux, let's disable this test on them.

Comment 11

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

Attachment: Bug 1925510 - Disable the profiler gtests on 32-bit platforms r?aabh!

Looks like Bug 1923595 changes how we calculate the number of cores and probably
resulting us to create more threads. Our GeckoProfiler.AllThreads test tries to
profile all the threads with some different all-thread features. It looks like
we hit the memory threshold on 32-bit Windows and Linux tests. This patch
disables it on these platforms so we don't get OOM intermittent failures.

Comment 12

[Intermittent Failures Robot]

16 failures in 868 pushes (0.018 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3

Platform and build breakdown:

• windows11-32-2009-qr: 13
  • opt: 13
• windows11-32-2009-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1925510&startday=2024-11-06&endday=2024-11-06&tree=all

Comment 13

[Nazım Can Altınova [:canova][:canaltinova on phabricator]]

As Bug 1919384 comment 29 shows, it looks like it's not just this subtest and more a general issue. So I'm disabling the gtests on all 32-bit platforms with this patch instead.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eeca264bb82b
Disable the profiler gtests on 32-bit platforms r=aabh,profiler-reviewers

Comment 15

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/eeca264bb82b

Comment 16

[Intermittent Failures Robot]

47 failures in 4515 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 40
• mozilla-central: 7

Platform and build breakdown:

• windows11-32-2009-qr: 40
  • opt: 40
• windows11-32-2009-shippable-qr: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1925510&startday=2024-11-04&endday=2024-11-10&tree=all