Closed Bug 1575773 Opened 5 years ago Closed 4 years ago

cpppunittests can be skipped/fail silently

Categories

(Testing :: CPPUnitTest, defect, P5)

Version 3
defect

Tracking

(firefox79 fixed)

RESOLVED FIXED
mozilla79
Tracking Status
firefox79 --- fixed

People

(Reporter: aryx, Assigned: gbrown)

References

Details

(Whiteboard: dev-prod-2020)

Attachments

(1 file)

Bug 1573051 accidentally removed a skip-if = os != 'win' rule for a Cpp test (moved it from TestNativeNt to TestUriValidation): https://hg.mozilla.org/mozilla-central/rev/4eca0f08c43b#l1.11

This got fixed in bug 1575655. In between, TestNativeNt should have run e.g. on Linux but it cannot be found in the log:

Formatted log: https://treeherder.mozilla.org/logviewer.html#?job_id=262411979&repo=mozilla-central
Plain text log: https://queue.taskcluster.net/v1/task/ZJTeZO8UR_KnaggrhWrugA/runs/0/artifacts/public/logs/live_backing.log

If the test compilation or execution fails, the task should report this as an error.

https://searchfox.org/mozilla-central/source/mozglue/tests/moz.build#26 determines if TestNativeNt should be built independently of testing/cppunittest.ini. Compilation failures would have appeared in the Build log not the test log.

https://searchfox.org/mozilla-central/source/testing/runcppunittests.py#266 silently drops tests specified in the manifest which do not exist. I suppose we could log these as warnings but I don't know how much noise there would be if we did so. As it stands, the manifest's exclusion of tests for platforms where the test is not built has no effect.

Priority: -- → P5
Whiteboard: dev-prod-2020

I suppose we could log these as warnings but I don't know how much noise there would be if we did so.

Let's find out!
https://treeherder.mozilla.org/#/jobs?repo=try&revision=86cf4569fdaf1d0ab40b998a78058286a9c4155c

All platforms have warnings! For linux/opt I see:

[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/TestAppShellSteadyState - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/TestParseFTPList - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/TestPLDHash - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/TestSyncRunnable - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/TestVolatileBuffer - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/buffered_stun_socket_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/ice_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/test_nr_socket_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/multi_tcp_socket_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/nrappkit_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/rlogringbuffer_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/runnable_utils_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/sctp_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/signaling_unittests - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/signaling_unittests_standalone - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/simpletokenbucket_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/sockettransportservice_unittest - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/transport_unittests - skipped
[task 2020-06-01T16:23:33.771Z] 16:23:33     INFO -  test file not found: /builds/worker/workspace/build/tests/cppunittest/turn_unittest - skipped
Assignee: nobody → gbrown

Most of these were switched to gtest; see 1239870, for instance. The remaining tests are:

TestAppShellSteadyState
signaling_unittests
signaling_unittests_standalone

See Also: → 1642666
See Also: → 1642686
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/94a6c07bd0d6
Warn when cppunit test file missing, and cleanup cppunit manifest; r=bc
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla79
See Also: → 1760857
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: