1745116 - Intermittent ThreadMetrics.CollectMetrics | Expected: (duration) >= (50000u), actual: 48922 vs 50000 @ /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestThreadMetrics.cpp:186

Status: RESOLVED FIXED

(Core :: XPCOM, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=360560748&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Ru1mw6q3QhiohaUWHwCqzQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-12-09T03:25:02.131Z] 03:25:02     INFO -  TEST-START | ThreadMetrics.CollectMetrics
[task 2021-12-09T03:25:02.628Z] 03:25:02  WARNING -  TEST-UNEXPECTED-FAIL | ThreadMetrics.CollectMetrics | Expected: (duration) >= (50000u), actual: 48922 vs 50000 @ /builds/worker/checkouts/gecko/xpcom/tests/gtest/TestThreadMetrics.cpp:186
[task 2021-12-09T03:25:02.631Z] 03:25:02  WARNING -  TEST-UNEXPECTED-FAIL | ThreadMetrics.CollectMetrics | test completed (time: 499ms)

Comment 1

[Intermittent Failures Robot]

5 failures in 3319 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• try: 2

Platform and build breakdown:

• windows10-64-2004-qr: 3
  • debug: 2
  • opt: 1
• windows10-64-2004-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2021-12-06&endday=2021-12-12&tree=all

Comment 2

[Iulian Moraru]

Update:
There have been 42 failures within the last 7 days:
• 13 failures on Windows 10 x64 2004 asan WebRender opt
• 3 failures on Windows 10 x64 2004 CCov WebRender opt
• 14 failures on Windows 10 x64 2004 WebRender debug
• 11 failures on Windows 10 x64 2004 WebRender opt
• 1 failures on Windows 10 x64 2004 WebRender Shippable opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=361823931&repo=mozilla-central

[task 2021-12-20T00:16:57.152Z] 00:16:57     INFO -  TEST-START | ThreadMetrics.CollectMetrics
[task 2021-12-20T00:16:57.244Z] 00:16:57  WARNING -  TEST-UNEXPECTED-FAIL | ThreadMetrics.CollectMetrics | Expected: (duration) >= (50000u), actual: 47275 vs 50000 @ Z:/task_163995083551877/build/src/xpcom/tests/gtest/TestThreadMetrics.cpp:186
[task 2021-12-20T00:16:57.257Z] 00:16:57  WARNING -  TEST-UNEXPECTED-FAIL | ThreadMetrics.CollectMetrics | test completed (time: 105ms)
[task 2021-12-20T00:16:57.257Z] 00:16:57     INFO -  TEST-START | ThreadMetrics.CollectRecursiveMetrics

Comment 3

[Iulian Moraru]

Hi Nika! Could you please help us assign this to someone?
Thank you!

Comment 4

[Intermittent Failures Robot]

45 failures in 3263 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #20 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 34
• mozilla-central: 8
• mozilla-esr91: 2
• try: 1

Platform and build breakdown:

• windows10-64-2004-asan-qr: 13
  • opt: 13
• windows10-64-2004-qr: 27
  • opt: 11
  • debug: 16
• windows10-64-2004-ccov-qr: 3
  • opt: 3
• windows10-64-2004-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2021-12-13&endday=2021-12-19&tree=all

Comment 5

[Intermittent Failures Robot]

51 failures in 2286 pushes (0.022 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 36
• try: 8
• mozilla-central: 7

Platform and build breakdown:

• windows10-64-2004-qr: 34
  • opt: 9
  • debug: 25
• windows10-64-2004-asan-qr: 10
  • opt: 10
• windows10-64-2004-shippable-qr: 4
  • opt: 4
• windows10-64-2004-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2021-12-20&endday=2021-12-26&tree=all

Comment 6

[Intermittent Failures Robot]

41 failures in 1011 pushes (0.041 failures/push) were associated with this bug in the last 7 days.

This is the #11 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 22
• mozilla-central: 17
• try: 2

Platform and build breakdown:

• windows10-64-2004-qr: 24
  • debug: 15
  • opt: 9
• windows10-64-2004-asan-qr: 11
  • opt: 11
• windows10-64-2004-shippable-qr: 3
  • opt: 3
• windows10-64-2004-ccov-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2021-12-27&endday=2022-01-02&tree=all

Comment 7

[Nika Layzell [:nika] (ni? for response)]

:bwc, could you perhaps look into this? I'm thinking this is probably caused by PR_Sleep waking us up slightly early and the sleeping code in TimedRunnable not accommodating for it. (https://searchfox.org/mozilla-central/rev/fac07284a9a996ddf968ea53adaf25c2a8b7c520/xpcom/tests/gtest/TestThreadMetrics.cpp#53-62)

Comment 8

[Atila Butkovits]

There have been 40 failures in the last 7 days: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2021-12-29&endday=2022-01-05&tree=trunk&bug=1745116

Happens on:
-windows10-64-2004-asan-qr opt
-windows10-64-2004-qr opt and debug.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=363053512&repo=autoland&lineNumber=4624

Comment 9

[Intermittent Failures Robot]

19 failures in 574 pushes (0.033 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 11
• mozilla-central: 8

Platform and build breakdown:

• windows10-64-2004-qr: 10
  • opt: 7
  • debug: 3
• windows10-64-2004-asan-qr: 3
  • opt: 3
• windows10-64-2004-ccov-qr: 4
  • opt: 4
• windows10-64-2004-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-01-05&endday=2022-01-05&tree=all

Comment 10

[Byron Campen [:bwc]]

Sorry, I have my plate full at the moment.

Comment 11

[Intermittent Failures Robot]

57 failures in 2801 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #8 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 36
• mozilla-central: 17
• mozilla-esr91: 2
• try: 2

Platform and build breakdown:

• windows10-64-2004-qr: 33
  • opt: 17
  • debug: 16
• windows10-64-2004-asan-qr: 13
  • opt: 13
• windows10-64-2004-ccov-qr: 7
  • opt: 7
• windows10-64-2004-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-01-03&endday=2022-01-09&tree=all

Comment 12

[Cristian Tuns]

Attachment: Bug 1745116 - disable ThreadMetrics.CollectMetrics on windows x64 r=#intermittent-reviewers

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/73f018a3325f
disable ThreadMetrics.CollectMetrics on windows x64 r=intermittent-reviewers,MasterWayZ DONTBUILD

Comment 14

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/73f018a3325f

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr91/rev/ab9feaedb93b5915ed1b4d2a23fe30c89f15fc60

Comment 16

[Intermittent Failures Robot]

8 failures in 3060 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2
• mozilla-beta: 5
• try: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 5
  • opt: 5
• windows10-64-2004-qr: 1
  • debug: 1
• windows10-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-01-10&endday=2022-01-16&tree=all

Comment 17

[Intermittent Failures Robot]

2 failures in 3444 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 2

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-01-17&endday=2022-01-23&tree=all

Comment 18

[Intermittent Failures Robot]

3 failures in 3445 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 3

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-01-24&endday=2022-01-30&tree=all

Comment 19

[Intermittent Failures Robot]

2 failures in 3711 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-02-07&endday=2022-02-13&tree=all

Comment 20

[Intermittent Failures Robot]

1 failures in 3063 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-02-28&endday=2022-03-06&tree=all

Comment 21

[Sandor Molnar]

This failure started to appear again after bug 1780614 landed
At the moment fails only on win10 x86 and looks high frequency

@Nika, can you take a look?

Comment 22

[Nika Layzell [:nika] (ni? for response)]

The test was already disabled on 64-bit windows previously in this bug due to high intermittent rates. It's somewhat unsurprising to me that it would start occurring on x86 as well after some changes nearby this code.

That being said, the particular bug is pretty evident here (sleep calls can end early, and the check doesn't take that into account. unlike all other tests in the file), so I'll post a patch.

Comment 23

[Nika Layzell [:nika] (ni? for response)]

Attachment: Bug 1745116 - Make TestThreadMetrics check against how much time was actually spent, r=smaug

Comment 24

[Intermittent Failures Robot]

30 failures in 3057 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

This is the #49 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 24
• mozilla-central: 4
• try: 2

Platform and build breakdown:

• windows10-32-2004-qr: 27
  • opt: 15
  • debug: 12
• windows10-32-2004-shippable-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-08-01&endday=2022-08-07&tree=all

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cf15683f9c21
Make TestThreadMetrics check against how much time was actually spent, r=smaug

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/1fc0584921f9

Comment 27

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/cf15683f9c21

Comment 28

[Intermittent Failures Robot]

8 failures in 3339 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 1
• try: 2

Platform and build breakdown:

• windows10-32-2004-qr: 6
  • debug: 4
  • opt: 2
• windows10-32-2004-shippable-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745116&startday=2022-08-08&endday=2022-08-14&tree=all