Closed Bug 951318 Opened 7 years ago Closed 7 years ago

TEST-UNEXPECTED-FAIL | /test/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js

Categories

(Thunderbird :: General, defect)

defect
Not set
normal

Tracking

(thunderbird27 fixed, thunderbird28 fixed)

RESOLVED FIXED
Thunderbird 29.0
Tracking Status
thunderbird27 --- fixed
thunderbird28 --- fixed

People

(Reporter: mkmelin, Assigned: standard8)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

All platforms.

https://tbpl.mozilla.org/php/getParsedLog.php?id=32096380&tree=Thunderbird-Trunk&full=1


TEST-INFO | /builds/slave/test/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js | running test ...
TEST-UNEXPECTED-FAIL | /builds/slave/test/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>
TEST-INFO | (xpcshell/head.js) | test MAIN run_test pending (1)
TEST-INFO | (xpcshell/head.js) | test run_next_test 0 pending (2)
TEST-INFO | (xpcshell/head.js) | test MAIN run_test finished (2)
TEST-INFO | (xpcshell/head.js) | running event loop
TEST-INFO | /builds/slave/test/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js | Starting test_expired_pings_are_deleted
TEST-INFO | (xpcshell/head.js) | test test_expired_pings_are_deleted pending (2)
TEST-INFO | (xpcshell/head.js) | test run_next_test 0 finished (2)
TEST-UNEXPECTED-FAIL | /builds/slave/test/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js | Found 5 unexpected saved pings. - See following stack:
JS frame :: /builds/slave/test/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js :: assertNotSaved :: line 160
JS frame :: /builds/slave/test/build/xpcshell/tests/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js :: test_expired_pings_are_deleted :: line 229
JS frame :: resource://gre/modules/Task.jsm :: TaskImpl_run :: line 233
JS frame :: resource://gre/modules/Promise.jsm :: Handler.prototype.process :: line 767
JS frame :: resource://gre/modules/Promise.jsm :: this.PromiseWalker.walkerLoop :: line 531
native frame :: <unknown filename> :: <TOP_LEVEL> :: line 0
TEST-INFO | (xpcshell/head.js) | exiting test
TEST-UNEXPECTED-FAIL | resource://gre/modules/commonjs/sdk/core/promise.js | Unexpected exception 2147500036
TEST-INFO | (xpcshell/head.js) | exiting test
!!! error running onStopped callback: TypeError: callback is not a function
<<<<<<<
Node SPDY server shutting down ...
INFO | Result summary:
INFO | Passed: 2042
INFO | Failed: 1
INFO | Todo: 4
INFO | Retried: 2
mxr/hg say bug 863872 added this test. Normally with these issues, we've got some sort of preference that's differently set, however I don't see that here.
Blocks: 863872
Indeed - I'm the one who added this test. I'll look into this.
Assignee: nobody → mconley
So I'm having no luck running the telemetry xpcshell tests locally.

Here's what I'm executing / getting:

Mikes-MacBook-Pro-2:objdir-tb mikeconley$ make xpcshell-tests TEST_PATH=toolkit/components/telemetry/tests
make -C mozilla xpcshell-tests
Have you considered running xpcshell tests via |mach xpcshell-test|? mach is easier to use and has more features than make and it will eventually be the only way to run xpcshell tests. Please consider using mach today!
/Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_virtualenv/bin/python -u /Users/mikeconley/Projects/comm-central/mozilla/config/pythonpath.py \
	  -I./build \
	  -I/Users/mikeconley/Projects/comm-central/mozilla/build \
	  -I./_tests/mozbase/mozinfo \
	  /Users/mikeconley/Projects/comm-central/mozilla/testing/xpcshell/runxpcshelltests.py \
	  --manifest=./_tests/xpcshell/xpcshell.ini \
	  --build-info-json=./mozinfo.json \
	  --no-logfiles \
	  --test-plugin-path='./dist/plugins' \
	  --tests-root-dir=/Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell \
	  --testing-modules-dir=/Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/modules \
	  --xunit-file=/Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/results.xml \
	  --xunit-suite-name=xpcshell \
          --symbols-path=./dist/crashreporter-symbols \
	  --test-path='toolkit/components/telemetry/tests'  \
	  /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/dist/bin/xpcshell
INFO | Using at most 32 threads.
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetryPing.js | running test ...
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetryStopwatch.js | running test ...
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetryLockCount.js | running test ...
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetryLateWrites.js | running test ...
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_nsITelemetry.js | running test ...
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetryPing_idle.js | running test ...
TEST-INFO | Failed to remove directory: /var/folders/bq/n5qs5f650bl7kmj4h_fqsw2m0000gn/T/tmpZj9nQW. Waiting.
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_ThirdPartyCookieProbe.js | running test ...
TEST-INFO | test_TelemetryLateWrites.js | Test failed or timed out, will retry.
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetryPingBuildID.js | running test ...
TEST-INFO | test_TelemetryLockCount.js | Test failed or timed out, will retry.
TEST-INFO | test_TelemetryPing.js | Test failed or timed out, will retry.
TEST-INFO | test_TelemetryPingBuildID.js | Test failed or timed out, will retry.
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetrySendOldPings.js | running test ...
TEST-INFO | test_TelemetryPing_idle.js | Test failed or timed out, will retry.
TEST-INFO | test_ThirdPartyCookieProbe.js | Test failed or timed out, will retry.
TEST-INFO | test_nsITelemetry.js | Test failed or timed out, will retry.
TEST-INFO | test_TelemetrySendOldPings.js | Test failed or timed out, will retry.
TEST-INFO | test_TelemetryStopwatch.js | Test failed or timed out, will retry.
Retrying tests that failed when run in parallel.
TEST-INFO | /Users/mikeconley/Projects/comm-central/objdir-tb/mozilla/_tests/xpcshell/toolkit/components/telemetry/tests/unit/test_TelemetryLockCount.js | running test ...
INFO | Following exceptions were raised:
Traceback (most recent call last):
  File "/Users/mikeconley/Projects/comm-central/mozilla/testing/xpcshell/runxpcshelltests.py", line 145, in run
    self.run_test()
  File "/Users/mikeconley/Projects/comm-central/mozilla/testing/xpcshell/runxpcshelltests.py", line 604, in run_test
    stdout=self.pStdout, stderr=self.pStderr, env=self.env, cwd=test_dir)
  File "/Users/mikeconley/Projects/comm-central/mozilla/testing/xpcshell/runxpcshelltests.py", line 224, in launchProcess
    env=env, cwd=cwd)
  File "/Users/mikeconley/Projects/comm-central/mozilla/python/psutil/psutil/__init__.py", line 855, in __init__
    self.__subproc = subprocess.Popen(*args, **kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
  File "/Users/mikeconley/Projects/comm-central/mozilla/config/pythonpath.py", line 56, in <module>
    main(sys.argv[1:])
  File "/Users/mikeconley/Projects/comm-central/mozilla/config/pythonpath.py", line 48, in main
    execfile(script, frozenglobals)
  File "/Users/mikeconley/Projects/comm-central/mozilla/testing/xpcshell/runxpcshelltests.py", line 1594, in <module>
    main()
  File "/Users/mikeconley/Projects/comm-central/mozilla/testing/xpcshell/runxpcshelltests.py", line 1590, in main
    if not xpcsh.runTests(args[0], testdirs=args[1:], **options.__dict__):
  File "/Users/mikeconley/Projects/comm-central/mozilla/testing/xpcshell/runxpcshelltests.py", line 1475, in runTests
    raise exceptions[0]
OSError: [Errno 2] No such file or directory
make[1]: *** [xpcshell-tests] Error 1
make: *** [xpcshell-tests] Error 2
Mikes-MacBook-Pro-2:objdir-tb mikeconley$


Any ideas how I can try running these tests locally?
Flags: needinfo?(mbanner)
make xpcshell-tests TEST_PATH=toolkit/components/telemetry/tests is working for me here.

I did have to change my mozconfig file to get it to build:

mk_add_options MOZ_OBJDIR=/Users/moztest/comm/main/tb

(using non-relative dir)
Flags: needinfo?(mbanner)
Ok, I've worked this out. This fails because Thunderbird doesn't define:

MOZ_TELEMETRY_REPORTING

This means that telemetry doesn't really get enabled, and possible handle any data during the tests. I think it also means that we're not sending information generally though I'm struggling to confirm that at the moment.
So the configuration variable was added ages ago, but bug 722240 actually introduced reliance on it for sending telemetry reports in Gecko 13.... so we've probably not been actually sending reports for quite a while :-(

This patch adds the configuration variable, and will get the tests passing, and we'll be sending results again.

It appears the dashboard doesn't support other apps currently, but I've filed an issue on it:

https://github.com/mozilla/telemetry-dashboard/issues/16

There's a try run with this patch here (windows disabled due to other issues):

https://tbpl.mozilla.org/?tree=Thunderbird-Try
Attachment #8354561 - Flags: review?(mconley)
Assignee: mconley → mbanner
Comment on attachment 8354561 [details] [diff] [review]
Enable Telemetry Reporting

Review of attachment 8354561 [details] [diff] [review]:
-----------------------------------------------------------------

Yes, this does the job! Thanks!
Attachment #8354561 - Flags: review?(mconley) → review+
Also, I figured out the XPCshell test issue - I had to alter my .mozconfig to place the objdir as a sibling of the srcdir (as opposed to being a child of the srcdir).
https://hg.mozilla.org/comm-central/rev/edb466c19b8b
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 29.0
Seems we're still hitting this on Windows on c-c tip?
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #22)
> Seems we're still hitting this on Windows on c-c tip?

I suspect these were old builds that hadn't been clobbered. Seems fine now.
Comment on attachment 8354561 [details] [diff] [review]
Enable Telemetry Reporting

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: 
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky):

[Triage Comment]

[Triage Comment]
Taking onto aurora and beta as bug 863872 is also there. Still thinking about esr.
Attachment #8354561 - Flags: approval-comm-esr24?
Attachment #8354561 - Flags: approval-comm-beta+
Attachment #8354561 - Flags: approval-comm-aurora+
Comment on attachment 8354561 [details] [diff] [review]
Enable Telemetry Reporting

Decided not to take this for esr, we'll pick it up in the next ESR cycle.
Attachment #8354561 - Flags: approval-comm-esr24? → approval-comm-esr24-
You need to log in before you can comment on or make changes to this bug.