Intermittent toolkit/components/telemetry/tests/unit/test_TelemetrySend.js | test_sendCheckOverride - [test_sendCheckOverride : 528] Must receive a ping of the expected type - "health" == "test-sendCheckOverride"
Categories
(Toolkit :: Telemetry, defect, P2)
Tracking
()
Tracking | Status | |
---|---|---|
firefox-esr68 | --- | disabled |
People
(Reporter: intermittent-bug-filer, Assigned: chutten)
References
(Regression)
Details
(Keywords: intermittent-failure, regression, Whiteboard: [retriggered][stockwell disabled])
Attachments
(2 files)
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 41•6 years ago
|
||
Failure reappeared on this merge https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=0e9e744accd3ed7b7b0ff2436904ed78a92bf9e2
Bryce is this from bug 1532391?
There are 18 total failures in the last 3 days on linux64-shippable opt.
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=245516267&repo=autoland&lineNumber=2203
[task 2019-05-09T11:36:45.090Z] 11:36:45 INFO - TEST-START | toolkit/components/telemetry/tests/unit/test_TelemetrySend.js
[task 2019-05-09T11:38:17.635Z] 11:38:17 WARNING - TEST-UNEXPECTED-FAIL | toolkit/components/telemetry/tests/unit/test_TelemetrySend.js | xpcshell return code: 0
[task 2019-05-09T11:38:17.636Z] 11:38:17 INFO - TEST-INFO took 92537ms
[task 2019-05-09T11:38:17.636Z] 11:38:17 INFO - >>>>>>>
[task 2019-05-09T11:38:17.640Z] 11:38:17 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2019-05-09T11:38:17.640Z] 11:38:17 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2019-05-09T11:38:17.642Z] 11:38:17 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2019-05-09T11:38:17.642Z] 11:38:17 INFO - running event loop
[task 2019-05-09T11:38:17.643Z] 11:38:17 INFO - toolkit/components/telemetry/tests/unit/test_TelemetrySend.js | Starting test_setup
[task 2019-05-09T11:38:19.433Z] 11:38:19 INFO - TEST-PASS | toolkit/components/telemetry/tests/unit/test_TelemetrySend.js | test_sendCheckOverride - [test_sendCheckOverride : 1] Known clientId should never appear in a ping on the server - "c0ffeec0-ffee-c0ff-eec0-ffeec0ffeec0" != "38178b0d-0302-409a-96e2-4241bfeb478c"
[task 2019-05-09T11:38:19.434Z] 11:38:19 WARNING - TEST-UNEXPECTED-FAIL | toolkit/components/telemetry/tests/unit/test_TelemetrySend.js | test_sendCheckOverride - [test_sendCheckOverride : 552] Must receive a ping of the expected type - "health" == "test-sendCheckOverride"
[task 2019-05-09T11:38:19.434Z] 11:38:19 INFO - /builds/worker/workspace/build/tests/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySend.js:test_sendCheckOverride:552
[task 2019-05-09T11:38:19.435Z] 11:38:19 INFO - exiting test
[task 2019-05-09T11:38:19.435Z] 11:38:19 INFO - Unexpected exception NS_ERROR_ABORT:
[task 2019-05-09T11:38:19.435Z] 11:38:19 INFO - _abort_failed_test@/builds/worker/workspace/build/tests/xpcshell/head.js:742:20
[task 2019-05-09T11:38:19.435Z] 11:38:19 INFO - do_report_result@/builds/worker/workspace/build/tests/xpcshell/head.js:849:5
[task 2019-05-09T11:38:19.436Z] 11:38:19 INFO - Assert<@/builds/worker/workspace/build/tests/xpcshell/head.js:57:5
[task 2019-05-09T11:38:19.436Z] 11:38:19 INFO - proto.report@resource://testing-common/Assert.jsm:213:10
[task 2019-05-09T11:38:19.436Z] 11:38:19 INFO - equal@resource://testing-common/Assert.jsm:249:8
[task 2019-05-09T11:38:19.437Z] 11:38:19 INFO - test_sendCheckOverride@/builds/worker/workspace/build/tests/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySend.js:552:10
[task 2019-05-09T11:38:19.437Z] 11:38:19 INFO - async*run_next_test/_run_next_test/<@/builds/worker/workspace/build/tests/xpcshell/head.js:1437:22
[task 2019-05-09T11:38:19.437Z] 11:38:19 INFO - _run_next_test@/builds/worker/workspace/build/tests/xpcshell/head.js:1437:38
[task 2019-05-09T11:38:19.438Z] 11:38:19 INFO - run@/builds/worker/workspace/build/tests/xpcshell/head.js:688:9
[task 2019-05-09T11:38:19.438Z] 11:38:19 INFO - _do_main@/builds/worker/workspace/build/tests/xpcshell/head.js:227:6
[task 2019-05-09T11:38:19.438Z] 11:38:19 INFO - _execute_test@/builds/worker/workspace/build/tests/xpcshell/head.js:529:5
[task 2019-05-09T11:38:19.439Z] 11:38:19 INFO - @-e:1:1
[task 2019-05-09T11:38:19.439Z] 11:38:19 INFO - exiting test
Updated•6 years ago
|
I suppose it could be given the timing. Bug 1548959 also landed in that range.
I'm not particularly familiar with this test, but we'd previously had the media recorder probes in place seemingly without issue. As bug 1532391 restored them after they lapsed, I'd be surprised if it introduced new bustages, but I could use some help from those more familiar with telem and this test. I'm going to hold the NI to remind me to come back to this and try and reach out for aid in understanding what this failure means.
Comment 43•6 years ago
|
||
Seems that this push is the culprit:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux64-shippable-qr%2Copt%2Cxpcshell%2Ctests%2Ctest-linux64-shippable-qr%2Fopt-xpcshell-e10s-7%2Cx%28x7%29&revision=90e7019e0fa8e062f4b887d2802e203e4bd39300&selectedJob=245906311
Mark, can you please take a look at this?
Updated•6 years ago
|
Comment 44•6 years ago
|
||
(In reply to Cristina Coroiu [:ccoroiu] from comment #43)
Mark, can you please take a look at this?
All bug 1548639/bug 1541416 did was change some unrelated test code and add some new tests. So directly they wouldn't have caused this.
I've just scanned the logs and it looks like test_TelemetrySend.js is now being run around the same time as the tests in bug 1548639 affected. Those tests are very processor intensive at the moment (I'm working on making them a bit better).
It looks like test_TelemetrySend.js is now failing/timing out when run in parallel, and then gets re-run sequentially as a result which is then failing.
I just ran it locally on my Mac (opt, artifact) as a single test, and out of four times it failed twice.
My suspicion is that there's something timing related here. Generally it gets away with it if run in parallel alongside tests that aren't very processor intensive, but run it on its own, or under heavy load, and there's something triggering the failure.
It is probably better for this to be looked at by the Telemetry team as they will know more about this test and the possible timing issues.
Comment hidden (Intermittent Failures Robot) |
Updated•6 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 61•6 years ago
|
||
There have been 66 failures within the last 7 days as it follows:
- 2 failures on linux x64 opt
- 1 failure on linux x64 quantumrender opt
- 17 failures on linux x64 shippable opt
- 28 failures on linux x64 quantumrender shippable opt
- 2 failures on os x 10.14 shippable opt
- 1 failure on windows 7 opt
- 15 failures on windows 7 shippable opt
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=260966673&repo=autoland&lineNumber=6272
[task 2019-08-10T20:22:30.924Z] 20:22:30 INFO - TEST-START | toolkit/components/telemetry/tests/unit/test_TelemetrySend.js
[task 2019-08-10T20:22:55.642Z] 20:22:55 WARNING - TEST-UNEXPECTED-FAIL | toolkit/components/telemetry/tests/unit/test_TelemetrySend.js | xpcshell return code: 0
[task 2019-08-10T20:22:55.642Z] 20:22:55 INFO - TEST-INFO took 24719ms
[task 2019-08-10T20:22:55.642Z] 20:22:55 INFO - >>>>>>>
Comment hidden (Intermittent Failures Robot) |
Comment 63•6 years ago
|
||
Comment 64•6 years ago
|
||
Updated•6 years ago
|
Comment 65•6 years ago
|
||
bugherder |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 68•6 years ago
|
||
Since the patch the only oranges are from beta, esr, inbound, and try. Looks like disabling on !debug did the trick.
I looked into a quick fix for this, but simply disabling the "health" ping for the test didn't help. I think maybe there's carry-over from an earlier test that's causing the problem and a proper level of clean-up would take care of things, but I haven't looked too deeply.
Assignee | ||
Updated•6 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 83•5 years ago
|
||
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 110•4 years ago
|
||
Comment 111•4 years ago
|
||
Comment 112•4 years ago
|
||
bugherder |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Updated•3 years ago
|
Assignee | ||
Updated•3 years ago
|
Updated•3 years ago
|
Updated•3 years ago
|
Description
•