1849415 - Intermittent FOGFixture.TestCppTimingDistWorks | Expected: (data.sum) > ((uint64_t)(15 * NANOS_IN_MILLIS) - EPSILON), actual: 10734000 vs 14800000 @ /builds/worker/checkouts/gecko/toolkit/components/glean/tests/gtest/TestFog.cpp:264

Status: RESOLVED INCOMPLETE

(Toolkit :: Telemetry, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=426514312&repo=mozilla-esr115
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cpYQ1Na5QgezNLhNTM47gQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-08-18T21:07:58.880Z] 21:07:58     INFO -  TEST-START | FOGFixture.TestCppTimingDistWorks
[task 2023-08-18T21:07:58.911Z] 21:07:58  WARNING -  TEST-UNEXPECTED-FAIL | FOGFixture.TestCppTimingDistWorks | Expected: (data.sum) > ((uint64_t)(15 * NANOS_IN_MILLIS) - EPSILON), actual: 10734000 vs 14800000 @ /builds/worker/checkouts/gecko/toolkit/components/glean/tests/gtest/TestFog.cpp:264
[task 2023-08-18T21:07:58.911Z] 21:07:58  WARNING -  TEST-UNEXPECTED-FAIL | FOGFixture.TestCppTimingDistWorks | test completed (time: 20ms)

Comment 1

[Intermittent Failures Robot]

1 failures in 3076 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 2

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 3

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=428785674&repo=mozilla-esr115

Comment 4

[Intermittent Failures Robot]

1 failures in 3337 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-09-11&endday=2023-09-17&tree=all

Comment 5

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 6

[Chris H-C :chutten]

As you can see from the See Also list, timing tests are tough. The most complete exploration of this particular failing is in https://bugzilla.mozilla.org/show_bug.cgi?id=1813167#c1:

---

The 15 * NANOS_IN_MILLIS is the assumed, reasonable sum of one timing sample that slept for two 5ms intervals plus one timing sample that slept for one 5ms interval.

HOWEVER, things are rarely (see the bugs in the See Also list for just how rare) that reasonable. Those 5ms intervals are only "roughly" 5ms long, and if each of them is shaved juuuuust enough, we can find ourselves at a sum that is 15ms - (3 * SHAVE). This is why we put in EPSILON: we hope that EPSILON > (3 * SHAVE) and simultaneously keep it as small as possible.

Alas. An EPSILON of 0.04ms is apparently not always enough. We need one of at least 0.1867ms.

(( 1.3% seems like an awful lot for a timer to be off... but remember, this is three compounding errors, so it averages to 0.41%. Still not great, but. ))

(( Also, we straight up use an EPSILON of 2ms in the JS versions of this test (most of that's due to rounding, but not all of it) ))

A decent fix for this would be to up EPSILON to 200000 or (uint64_t)(0.2 * NANOS_IN_MILLIS)

---

If this comes back, we may need to re-examine whether we're correctly accounting for the errors in the three measurements or whether we're missing something.

Comment 7

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=432435691&repo=mozilla-esr115

Comment 8

[Intermittent Failures Robot]

1 failures in 3764 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 9

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 10

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=437858631&repo=mozilla-esr115

Comment 11

[Intermittent Failures Robot]

1 failures in 4108 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 12

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 13

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=442336395&repo=mozilla-esr115

Comment 14

[Intermittent Failures Robot]

1 failures in 2274 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 15

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 16

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=446723247&repo=mozilla-esr115

Comment 17

[Intermittent Failures Robot]

1 failures in 4009 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 18

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 19

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=487284362&repo=mozilla-esr115

Comment 20

[Intermittent Failures Robot]

Summary

1 failures in 3774 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

Table

	1proc	1proc-s	1proc-swr	a11y-checks	a11y-checks-s	a11y-checks-swr	a11y-checks-swr-s	condprof-s	fis	fis-hv	fis-hv-s	fis-s	headless	headless-s	http2	http3	mda-gpu	mda-gpu-s	msix	msix-s	no_variant	nofis	nofis-aab	nofis-s	nofis-ship	nofis-ship-s	nofis-spi	nofis-swr	nofis-swr-s	nogpu	nogpu-s	s	spi	spi-nw	spi-nw-1proc	spi-nw-nofis	spi-nw-nofis-s	spi-nw-s	spi-s	swr	swr-s	vt	vt-s	wmfme	wmfme-s	wr-dc0	wr-dc0-s	wr-dc1-p	wr-dc1-p-s	wr-dc2-o	wr-dc3-c	wr-dc3-c-s	xorig	xorig-s
windows7-32-shippable-qr/opt		1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2024-12-16&endday=2024-12-22&tree=all

Comment 21

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 22

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=493703468&repo=mozilla-esr115

Comment 23

[Intermittent Failures Robot]

Summary

1 failures in 4309 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-qr: 1
  • opt: 1

Table

	1proc
windows7-32-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2025-02-03&endday=2025-02-09&tree=all

Comment 24

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 25

[Intermittent Failures Robot]

Summary

1 failures in 4826 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-qr: 1
  • opt: 1

Table

	1proc
windows7-32-qr--/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2025-03-10&endday=2025-03-16&tree=all

Comment 26

[Intermittent Failures Robot]

2 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 2

Table

	1proc
windows7-32-qr--/opt	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2025-04-14&endday=2025-04-20&tree=all

Comment 27

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=508626774&repo=mozilla-esr115

Comment 28

[Intermittent Failures Robot]

3 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 3

Table

	1proc
windows7-x86/opt	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2025-05-12&endday=2025-05-18&tree=all

Comment 29

[BugBot [:suhaib / :marco]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 30

[Intermittent Failures Robot]

2 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 2

Table

	1proc
windows7-x86/opt	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2025-06-16&endday=2025-06-22&tree=all

Comment 31

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Table

	1proc
windows7-x86/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2025-12-08&endday=2025-12-14&tree=all

Comment 32

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Table

	1proc
windows7-x86/opt	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2026-01-05&endday=2026-01-11&tree=all