1606922 - stop packaging tests on Android platforms where we don't run tests

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Nathan Froyd [:froydnj]]

This is good for about 3-5% faster Android builds.

Comment 1

[Michael Shal [:mshal]]

I started to tackle this bug and bug 1606938 by manually going through mozconfigs, and realizing it is a painful process. I think it would be much better if the tests were only packaged for build tasks that actually have test tasks associated with them in the taskgraph. After chatting with tomprince about different approaches, we settled on manually setting MOZ_AUTOMATION_PACKAGE_TESTS in the task definitions, and adding a verification check to ensure it's set only for build tasks with associated tests.

The result is that we no longer set MOZ_AUTOMATION_PACKAGE_TESTS in mozconfigs, and the taskgraph check ensures that the flag is up-to-date in relevant build tasks as tests are added/removed.

Comment 2

[Michael Shal [:mshal]]

Attachment: Bug 1606922 - Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster; r?tomprince,#firefox-build-system-reviewers

We don't need to package tests for builds that we don't actually run
tests from, but it is tricky to align this correctly by setting
MOZ_AUTOMATION_PACKAGE_TESTS=0 in relevant mozconfigs. Instead we can
set the environment variable in the task definition, and use a full
taskgraph verification check to ensure that the flag is only set on
builds that have tests.

The one tricky task is win64-aarch64-shippable/opt, which copies tests
from another build rather than building them itself. For this reason, it
explicitly sets MOZ_AUTOMATION_PACKAGE_TESTS: '0' in the environment
even though that is now the default. This is why the exception is only
raised if MOZ_AUTOMATION_PACKAGE_TESTS is not set at all, rather than
checking that it is set to 1.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dd2f16711aba
Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster; r=firefox-build-system-reviewers,tomprince,chmanchester

Comment 4

[Daniel Varga [:dvarga]]

Backed out changeset dd2f16711aba (Bug 1606922) for causing build failures

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=dd2f16711abad9c23d42645954d8a6205f1ec14f

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284488213&repo=autoland&lineNumber=1033

Backout log: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=ca7b5c769effca1759b1085c2841c54a3446e825

[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  Error running mach:
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -      ['--log-no-times', 'artifact', 'install', '--symbols', '--host-bins']
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  The error occurred in code that was called by the mach command. This is either
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  a bug in the called code itself or in the way that mach is calling it.
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  isn't, please use |./mach busted file| to report it. If |./mach busted| is
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  If filing a bug, please include the full output of mach, including this error
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  message.
[task 2020-01-11T00:26:54.678Z] 00:26:54     INFO -  The details of the failure are as follows:
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -  ValueError: Expected tests archive matching "<_sre.SRE_Pattern object at 0x000000000386FB20>", but found none!
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -    File "z:/task_1578700043/build/src\python/mozbuild/mozbuild/artifact_commands.py", line 143, in artifact_install
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -      return artifacts.install_from(source, distdir or self.distdir)
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -    File "z:/task_1578700043/build/src\python/mozbuild\mozbuild\artifacts.py", line 1267, in install_from
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -      return self.install_from_task(os.environ[var], distdir)
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -    File "z:/task_1578700043/build/src\python/mozbuild\mozbuild\artifacts.py", line 1237, in install_from_task
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -      for artifact_name in self._artifact_job.find_candidate_artifacts(artifacts):
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -    File "z:/task_1578700043/build/src\python/mozbuild\mozbuild\artifacts.py", line 193, in find_candidate_artifacts
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -      'found none!'.format(re=self._tests_re))
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -  Makefile:139: recipe for target 'recurse_artifact' failed
[task 2020-01-11T00:26:54.679Z] 00:26:54     INFO -  mozmake.EXE[3]: *** [recurse_artifact] Error 1
[task 2020-01-11T00:26:54.680Z] 00:26:54     INFO -  mozmake.EXE[3]: Leaving directory 'z:/task_1578700043/build/src/obj-firefox'
[task 2020-01-11T00:26:54.680Z] 00:26:54     INFO -  z:/task_1578700043/build/src/config/recurse.mk:32: recipe for target 'artifact' failed
[task 2020-01-11T00:26:54.680Z] 00:26:54     INFO -  mozmake.EXE[2]: *** [artifact] Error 2
[task 2020-01-11T00:26:54.680Z] 00:26:54     INFO -  z:/task_1578700043/build/src/config/rules.mk:394: recipe for target 'default' failed
[task 2020-01-11T00:26:54.680Z] 00:26:54     INFO -  mozmake.EXE[1]: *** [default] Error 2
[task 2020-01-11T00:26:54.680Z] 00:26:54     INFO -  client.mk:125: recipe for target 'build' failed
[task 2020-01-11T00:26:54.680Z] 00:26:54     INFO -  mozmake.EXE: *** [build] Error 2
[task 2020-01-11T00:26:54.711Z] 00:26:54     INFO -  0 compiler warnings present.
[task 2020-01-11T00:26:54.796Z] 00:26:54    ERROR - Return code: 2
[task 2020-01-11T00:26:54.796Z] 00:26:54  WARNING - setting return code to 2
[task 2020-01-11T00:26:54.796Z] 00:26:54    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2020-01-11T00:26:54.796Z] 00:26:54    FATAL - Running post_fatal callback...
[task 2020-01-11T00:26:54.796Z] 00:26:54    FATAL - Exiting -1
[task 2020-01-11T00:26:54.796Z] 00:26:54     INFO - [mozharness: 2020-01-11 00:26:54.796000Z] Finished build step (failed)
[task 2020-01-11T00:26:54.796Z] 00:26:54     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2020-01-11T00:26:54.797Z] 00:26:54     INFO - Running command: ['C:\\mozilla-build\\msys\\bin\\bash.exe', 'z:\\task_1578700043\\build\\src\\mach', 'python', 'testing\\parse_build_tests_ccov.py'] in z:\task_1578700043\build\src
[task 2020-01-11T00:26:54.797Z] 00:26:54     INFO - Copy/paste: C:\mozilla-build\msys\bin\bash.exe z:\task_1578700043\build\src\mach python testing\parse_build_tests_ccov.py
[task 2020-01-11T00:26:54.801Z] 00:26:54     INFO - Using env: {'ACCEPTED_MAR_CHANNEL_IDS': 'firefox-mozilla-central',

Comment 6

[Michael Shal [:mshal]]

I updated the patch in phabricator with better support for the win64 aarch64 builds.

Comment 7

[Jordan Lund (:jlund)]

Tom has been out and will return next week. He can look at this review again then.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bc9bf388d476
Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster; r=tomprince,chmanchester

Comment 9

[Andreea Pavel [:apavel]]

Backed out for failing gecko decision task

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=bc9bf388d476205c4bc7afe402cf5c71e99cb7cc&selectedJob=290407931

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=290407931&repo=autoland&lineNumber=5097

[task 2020-02-25T17:51:32.306Z] Generating full task graph
[task 2020-02-25T17:51:32.349Z] Full task graph contains 17705 tasks and 58498 dependencies
[task 2020-02-25T17:51:36.131Z] Traceback (most recent call last):
[task 2020-02-25T17:51:36.131Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 194, in taskgraph_decision
[task 2020-02-25T17:51:36.131Z] return taskgraph.decision.taskgraph_decision(options)
[task 2020-02-25T17:51:36.131Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 210, in taskgraph_decision
[task 2020-02-25T17:51:36.131Z] full_task_json = tgg.full_task_graph.to_json()
[task 2020-02-25T17:51:36.131Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 153, in full_task_graph
[task 2020-02-25T17:51:36.131Z] return self._run_until('full_task_graph')
[task 2020-02-25T17:51:36.131Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 358, in _run_until
[task 2020-02-25T17:51:36.131Z] k, v = next(self._run)
[task 2020-02-25T17:51:36.131Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 291, in _run
[task 2020-02-25T17:51:36.131Z] yield verifications('full_task_graph', full_task_graph, graph_config)
[task 2020-02-25T17:51:36.132Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/verify.py", line 37, in call
[task 2020-02-25T17:51:36.132Z] verification(None, graph, scratch_pad=scratch_pad, graph_config=graph_config)
[task 2020-02-25T17:51:36.132Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/verify.py", line 288, in verify_test_packaging
[task 2020-02-25T17:51:36.132Z] 'MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment'.format(task.label))
[task 2020-02-25T17:51:36.132Z] Exception: Build job build-linux64-tsan/opt has tests dependent on it and does not specify MOZ_AUTOMATION_PACKAGE_TESTS=1 in the environment
[taskcluster 2020-02-25 17:51:37.027Z] === Task Finished ===
[taskcluster 2020-02-25 17:51:38.241Z] Unsuccessful task run with exit code: 1 completed in 72.511 seconds

Backout: https://hg.mozilla.org/integration/autoland/rev/5a4873d6a73b3427affe8d64457f7517b7594370

Comment 10

[Michael Shal [:mshal]]

Ahh, looks like linux64-tsan and macosx64-ccov both now have tests assigned to them. They just need the environment variable set - I'll try to re-land with that.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b7a8b4961e01
Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster; r=tomprince,chmanchester

Comment 12

[Marco Castelluccio [:marco]]

(In reply to Michael Shal [:mshal] from comment #10)

Ahh, looks like linux64-tsan and macosx64-ccov both now have tests assigned to them. They just need the environment variable set - I'll try to re-land with that.

FWIW, we don't schedule tests on macosx64-ccov yet (we might at some point).

Comment 13

[Michael Shal [:mshal]]

(In reply to Marco Castelluccio [:marco] from comment #12)

(In reply to Michael Shal [:mshal] from comment #10)

Ahh, looks like linux64-tsan and macosx64-ccov both now have tests assigned to them. They just need the environment variable set - I'll try to re-land with that.

FWIW, we don't schedule tests on macosx64-ccov yet (we might at some point).

There are test tasks that are generated in the taskgraph that depend on it though, like test-macosx1014-64-ccov/opt-cppunit-1proc and others, even if they don't run in autoland or central yet. The verification check added by this bug is just making sure that if there are test tasks that depend on a build, then the build has to package tests.

Comment 14

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/b7a8b4961e01

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/f666055d9525
Port bug 1606922: Move MOZ_AUTOMATION_PACKAGE_TESTS from mozconfigs to taskcluster. rs=bustage-fix