1813167 - Intermittent FOG.TestCppTimingDistWorks | Expected: (data.sum) > ((uint64_t)(15 * NANOS_IN_MILLIS) - EPSILON), actual: 14813300 vs 14960000 @ /builds/worker/checkouts/gecko/toolkit/components/glean/tests/gtest/TestFog.cpp:294

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=403807246&repo=mozilla-esr102
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/bpu5xbUJSSyfTS7lR_SFoA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2023-01-27T20:05:48.787Z] 20:05:48     INFO -  TEST-START | FOG.TestCppTimingDistWorks
[task 2023-01-27T20:05:48.797Z] 20:05:48  WARNING -  TEST-UNEXPECTED-FAIL | FOG.TestCppTimingDistWorks | Expected: (data.sum) > ((uint64_t)(15 * NANOS_IN_MILLIS) - EPSILON), actual: 14813300 vs 14960000 @ /builds/worker/checkouts/gecko/toolkit/components/glean/tests/gtest/TestFog.cpp:294
[task 2023-01-27T20:05:48.797Z] 20:05:48  WARNING -  TEST-UNEXPECTED-FAIL | FOG.TestCppTimingDistWorks | test completed (time: 10ms)
[task 2023-01-27T20:05:48.797Z] 20:05:48     INFO -  TEST-START | FOG.TestLabeledBooleanWorks

Comment 1

[Chris H-C :chutten]

The 15 * NANOS_IN_MILLIS is the assumed, reasonable sum of one timing sample that slept for two 5ms intervals plus one timing sample that slept for one 5ms interval.

HOWEVER, things are rarely (see the bugs in the See Also list for just how rare) that reasonable. Those 5ms intervals are only "roughly" 5ms long, and if each of them is shaved juuuuust enough, we can find ourselves at a sum that is 15ms - (3 * SHAVE). This is why we put in EPSILON: we hope that EPSILON > (3 * SHAVE) and simultaneously keep it as small as possible.

Alas. An EPSILON of 0.04ms is apparently not always enough. We need one of at least 0.1867ms.

(( 1.3% seems like an awful lot for a timer to be off... but remember, this is three compounding errors, so it averages to 0.41%. Still not great, but. ))

(( Also, we straight up use an EPSILON of 2ms in the JS versions of this test (most of that's due to rounding, but not all of it) ))

A decent fix for this would be to up EPSILON to 200000 or (uint64_t)(0.2 * NANOS_IN_MILLIS)

Comment 2

[Intermittent Failures Robot]

1 failures in 3914 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1813167&startday=2023-01-23&endday=2023-01-29&tree=all

Comment 3

[Razvan Cojocaru]

Attachment: Bug 1813167 - Up EPSILON to 200000 (0.2ms). r?chutten

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1d5957cb0ce5
Up EPSILON to 200000 (0.2ms). r=chutten

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/1d5957cb0ce5