1849415 - Intermittent FOGFixture.TestCppTimingDistWorks | Expected: (data.sum) > ((uint64_t)(15 * NANOS_IN_MILLIS) - EPSILON), actual: 10734000 vs 14800000 @ /builds/worker/checkouts/gecko/toolkit/components/glean/tests/gtest/TestFog.cpp:264

Status: RESOLVED INCOMPLETE

(Toolkit :: Telemetry, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=426514312&repo=mozilla-esr115
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cpYQ1Na5QgezNLhNTM47gQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-08-18T21:07:58.880Z] 21:07:58     INFO -  TEST-START | FOGFixture.TestCppTimingDistWorks
[task 2023-08-18T21:07:58.911Z] 21:07:58  WARNING -  TEST-UNEXPECTED-FAIL | FOGFixture.TestCppTimingDistWorks | Expected: (data.sum) > ((uint64_t)(15 * NANOS_IN_MILLIS) - EPSILON), actual: 10734000 vs 14800000 @ /builds/worker/checkouts/gecko/toolkit/components/glean/tests/gtest/TestFog.cpp:264
[task 2023-08-18T21:07:58.911Z] 21:07:58  WARNING -  TEST-UNEXPECTED-FAIL | FOGFixture.TestCppTimingDistWorks | test completed (time: 20ms)

Comment 1

[Intermittent Failures Robot]

1 failures in 3076 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-08-14&endday=2023-08-20&tree=all

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 3

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=428785674&repo=mozilla-esr115

Comment 4

[Intermittent Failures Robot]

1 failures in 3337 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-09-11&endday=2023-09-17&tree=all

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 6

[Chris H-C :chutten]

As you can see from the See Also list, timing tests are tough. The most complete exploration of this particular failing is in https://bugzilla.mozilla.org/show_bug.cgi?id=1813167#c1:

---

The 15 * NANOS_IN_MILLIS is the assumed, reasonable sum of one timing sample that slept for two 5ms intervals plus one timing sample that slept for one 5ms interval.

HOWEVER, things are rarely (see the bugs in the See Also list for just how rare) that reasonable. Those 5ms intervals are only "roughly" 5ms long, and if each of them is shaved juuuuust enough, we can find ourselves at a sum that is 15ms - (3 * SHAVE). This is why we put in EPSILON: we hope that EPSILON > (3 * SHAVE) and simultaneously keep it as small as possible.

Alas. An EPSILON of 0.04ms is apparently not always enough. We need one of at least 0.1867ms.

(( 1.3% seems like an awful lot for a timer to be off... but remember, this is three compounding errors, so it averages to 0.41%. Still not great, but. ))

(( Also, we straight up use an EPSILON of 2ms in the JS versions of this test (most of that's due to rounding, but not all of it) ))

A decent fix for this would be to up EPSILON to 200000 or (uint64_t)(0.2 * NANOS_IN_MILLIS)

---

If this comes back, we may need to re-examine whether we're correctly accounting for the errors in the three measurements or whether we're missing something.

Comment 7

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=432435691&repo=mozilla-esr115

Comment 8

[Intermittent Failures Robot]

1 failures in 3764 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-10-09&endday=2023-10-15&tree=all

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 10

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=437858631&repo=mozilla-esr115

Comment 11

[Intermittent Failures Robot]

1 failures in 4108 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 13

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=442336395&repo=mozilla-esr115

Comment 14

[Intermittent Failures Robot]

1 failures in 2274 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 16

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=446723247&repo=mozilla-esr115

Comment 17

[Intermittent Failures Robot]

1 failures in 4009 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 1

Platform and build breakdown:

• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1849415&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.