1525373 - Use run-task wrapper everywhere

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Chris AtLee [:catlee]]

At the moment, Windows and macOS tests aren't using run-task, which means they don't get nice timestamped logging.

I think we should try and make sure all tasks run under run-task.

Comment 1

[Connor Sheehan [:sheehan]]

In bug 1534463 I patched run-task to enable Mercurial operations to run against a private web head dedicated to CI. As part of testing that change on try, I discovered this bug. I might end up picking this up in the next little while, so we have more tasks performing checkouts against the private service, and so I don't need to re-implement that logic in various places.

Comment 3

[Mike Hommey [:glandium]]

Attachment: Bug 1525373 - Use the full path to chmod when setting up run-task.

The generic-worker tasks on bitbar, for some reason, don't have a PATH
set at all, and have it set via /builds/taskcluster/script.py. But the
command that sets run-task up runs outside that, so we have to be
explicit about where to find chmod.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1525373 - Use an explicit path for python2.7 for mozharness.

The macos workers have two python 2.7 installed: one in /usr/bin, and
one in /usr/local/bin. For some reason, the one in /usr/local/bin is
broken wrt SSL.

When running the current mozharness command directly without a full path
to python2.7, generic-worker executes it using its own PATH, and that
uses /usr/bin/python2.7. When wrapping with something else, though
(run-task, but that would apply just as much with sh -c), the PATH
from the task itself is used, and it's explicitly set to use
/usr/local/bin first, and the broken python 2.7 then gets used.

We could change the PATH, but there might be other things relying on the
current order, so just use /usr/bin/python2.7.

Depends on D28023

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1525373 - Add an option to `using: run-task` to keep running tasks as root.

The tasks on bitbar currently rely on being run as root, and run-task
defaults to drop its privileges to the worker user.

Depends on D28024

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1525373 - Use run_task transform for mozharness on docker-worker.

mozharness and mozharness test transforms currently do their own version
of wrapping commands with run-task in docker-worker. This makes them
piggy back on the run_task transform instead.

Some things then become redundant with what the run_task transform does,
and some others need to happen later than they currently do in order to
work.

This has the side effect of enabling some of the more recent run-task
features, like --fetch-hgfingerprint.

Depends on D28025

Comment 7

[Mike Hommey [:glandium]]

Attachment: Bug 1525373 - Use run-task for mozharness jobs on generic-worker.

mozharness and mozharness test transforms for generic-worker currently
don't wrap the commands with run-task. This changes things such that the
commands are wrapped with run-task, by piggy-backing on the run_task
transform.

Some things then become redundant with what the run_task transform does,
and some others need to happen later than they currently do in order to
work.

Depends on D28026

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1525373 - Work around bug 1551789.

TASKCLUSTER_ROOT_URL is not set on gecko-t-win10-64-hw workers, so set it
via transforms.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1fa5952696ca
Use an explicit path for python2.7 for mozharness. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/37f758cfb92a
Add an option to `using: run-task` to keep running tasks as root. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/00c0dfc00b1b
Use run_task transform for mozharness on docker-worker. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/108a519aff66
Use run-task for mozharness jobs on generic-worker. r=tomprince

Comment 10

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out by request.

Backout link: https://hg.mozilla.org/integration/autoland/rev/2fa3c884b6de3c07b3ff3b9736d7bedd5d38740c

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3e066ee9fbca
Use an explicit path for python2.7 for mozharness. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/58087407b5b9
Add an option to `using: run-task` to keep running tasks as root. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/447b8da8ebb3
Use run_task transform for mozharness on docker-worker. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/43cece5c03b4
Use run-task for mozharness jobs on generic-worker. r=tomprince

Comment 12

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 43cece5c03b4 (Bug 1525373) for Windows 10 AArch failures

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C10%2Caarch64%2Copt%2Ctest-windows10-aarch64%2Fopt-cppunit-1proc%2C%28cpp%29&fromchange=fb6f02f359d7883555d3c3590f18f71f484aa16e&tochange=9183f80b6bf0d03510336ac8faa0b98f50b56e9f&selectedJob=250538476

Backout link: https://hg.mozilla.org/integration/autoland/rev/9183f80b6bf0d03510336ac8faa0b98f50b56e9f

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250538476&repo=autoland&lineNumber=40

[taskcluster 2019-06-07T03:43:54.496Z] Task ID: PX9Vk69hSfOYv4GF4SOJUg
[taskcluster 2019-06-07T03:43:54.496Z] === Task Starting ===
[taskcluster 2019-06-07T03:43:55.447Z] Uploading redirect artifact public/logs/live.log to URL http://1.1.1.1:49940/log/C3fiq8_oR1aR42KfAjzYcQ with mime type "text/plain; charset=utf-8" and expiry 2019-06-07T04:58:55.029Z
[taskcluster 2019-06-07T03:43:55.821Z] [mounts] Downloading task PmX1rNt0S7WJbay2Gv2WEQ artifact public/build/mozharness.zip to downloads\JeX8zpXwRGG80dm4pOWWcw
[taskcluster 2019-06-07T03:43:56.495Z] [mounts] Downloaded 2342280 bytes with SHA256 2a766b4de461228351527418f0801051f72d83e58c5ab2811b4922b0d3a3540e from task PmX1rNt0S7WJbay2Gv2WEQ artifact public/build/mozharness.zip to downloads\JeX8zpXwRGG80dm4pOWWcw
[taskcluster:warn 2019-06-07T03:43:56.496Z] [mounts] Download downloads\JeX8zpXwRGG80dm4pOWWcw of task PmX1rNt0S7WJbay2Gv2WEQ artifact public/build/mozharness.zip has SHA256 2a766b4de461228351527418f0801051f72d83e58c5ab2811b4922b0d3a3540e but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2019-06-07T03:43:56.496Z] [mounts] Creating directory C:\tasks\task_1559877973 with permissions 0700
[taskcluster 2019-06-07T03:43:56.496Z] [mounts] Extracting zip file downloads\JeX8zpXwRGG80dm4pOWWcw to 'C:\tasks\task_1559877973'
[taskcluster 2019-06-07T03:43:57.012Z] [mounts] Downloading url https://hg.mozilla.org/integration/autoland/raw-file/43cece5c03b4ff1d1400a5be6b5fa255e8fcd8bf/taskcluster/scripts/run-task to downloads\Njc52SgMTwSjMtJ8zWrqsw
[taskcluster 2019-06-07T03:43:57.709Z] [mounts] Downloaded 30921 bytes with SHA256 8be03508dc6d71e4397d7d13eafb8927ebc47c36d5420fd12034c72fc6a50d30 from url https://hg.mozilla.org/integration/autoland/raw-file/43cece5c03b4ff1d1400a5be6b5fa255e8fcd8bf/taskcluster/scripts/run-task to downloads\Njc52SgMTwSjMtJ8zWrqsw
[taskcluster:warn 2019-06-07T03:43:57.709Z] [mounts] Download downloads\Njc52SgMTwSjMtJ8zWrqsw of url https://hg.mozilla.org/integration/autoland/raw-file/43cece5c03b4ff1d1400a5be6b5fa255e8fcd8bf/taskcluster/scripts/run-task has SHA256 8be03508dc6d71e4397d7d13eafb8927ebc47c36d5420fd12034c72fc6a50d30 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2019-06-07T03:43:57.709Z] [mounts] Creating directory C:\tasks\task_1559877973 with permissions 0700
[taskcluster 2019-06-07T03:43:57.710Z] [mounts] Copying downloads\Njc52SgMTwSjMtJ8zWrqsw to C:\tasks\task_1559877973\run-task
[taskcluster 2019-06-07T03:43:58.029Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --cppunittest-suite=cppunittest --disable-e10s --download-symbols ondemand --cppunittest-suite=cppunittest --disable-e10s
C:\tasks\task_1559877973>C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --cppunittest-suite=cppunittest --disable-e10s --download-symbols ondemand --cppunittest-suite=cppunittest --disable-e10s
[taskcluster 2019-06-07T03:43:58.334Z] Exit Code: -1073741515
[taskcluster 2019-06-07T03:43:58.334Z] User Time: 15.625ms
[taskcluster 2019-06-07T03:43:58.334Z] Kernel Time: 78.125ms
[taskcluster 2019-06-07T03:43:58.334Z] Wall Time: 298.0019ms
[taskcluster 2019-06-07T03:43:58.334Z] Result: FAILED
[taskcluster 2019-06-07T03:43:58.334Z] === Task Finished ===
[taskcluster 2019-06-07T03:43:58.334Z] Task Duration: 305.0053ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\tasks\task_1559877973\logs'", reason "file-missing-on-worker" and expiry 2020-06-06T03:30:42.803Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory 'C:\tasks\task_1559877973\build\blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2020-06-06T03:30:42.803Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\build\blobber_upload_dir'
[taskcluster 2019-06-07T03:43:59.645Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/PX9Vk69hSfOYv4GF4SOJUg/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-06-06T03:30:42.803Z
[taskcluster:error] exit status -1073741515
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\logs'
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\build\blobber_upload_dir'

Comment 13

[Mike Hommey [:glandium]]

Bug 1557614 is what caused those problems.

Comment 14

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/3e066ee9fbca
https://hg.mozilla.org/mozilla-central/rev/58087407b5b9
https://hg.mozilla.org/mozilla-central/rev/447b8da8ebb3

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d0c22c47c7f
Use run-task for mozharness jobs on generic-worker. r=tomprince

Comment 16

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/4d0c22c47c7f

Comment 17

[Rob Lemley [:rjl]]

Attachment: Bug 1525373 - Support comm-checkout when using generic-worker with run-task. r?#build

Fixes Thunderbird builds on Windows.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ba6240f6bc90
Support comm-checkout when using generic-worker with run-task. r=glandium

Comment 19

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/ba6240f6bc90

Comment 21

[Cosmin Sabou [:CosminS]]

Mike, looks like this introduced these raptor speedometer timeouts on Android: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258681723&repo=autoland&lineNumber=1118
https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=258681723&searchStr=raptor-speedometer-geckoview-power-e10s%2Crap%28sp-p%29&tochange=657c0163535f068a62447fa3e5f504c829bb15c8&fromchange=3eb04121209159cec75c59e0b1836ef8cad2435f
They stayed in tree after the backout of Bug 1561939 https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=c2a690d8461ca420f0ac9e864c07157bd89ea52a&selectedJob=258681721&searchStr=android%2C8.0%2Cpixel2%2Cpgo%2Craptor%2Cperformance%2Ctests%2Con%2Cfirefox%2Ctest-android-hw-p2-8-0-arm7-api-16%2Fpgo-raptor-speedometer-geckoview-power-e10s%2Crap%28sp-p%29
Could you please take a look at these?

A backout of this bug leads to green jobs: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=8c9a8fab632b2002a34fba4a3a4ad8ddc9714455&selectedJob=258684512

Comment 22

[Cosmin Sabou [:CosminS]]

Filed bug 1569459 to track those raptor failures.

Comment 23

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset ba6240f6bc90 (Bug 1525373) for speedometer timeouts

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=5805cd9ae2947386263069e1a3b2d832384bd45f&searchStr=android%2C8.0%2Cpixel2%2Cpgo%2Craptor%2Cperformance%2Ctests%2Con%2Cfirefox%2Ctest-android-hw-p2-8-0-arm7-api-16%2Fpgo-raptor-speedometer-geckoview-power-e10s%2Crap%28sp-p%29&selectedJob=258660767

Backout link: https://hg.mozilla.org/integration/autoland/rev/58f4b1223d19f8ea36d7aeb0bdb066654c0e96a4

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258660767&repo=mozilla-central&lineNumber=1599

Comment 24

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/58f4b1223d19

Comment 25

[Mike Hommey [:glandium]]

Attachment: Fixup bug 1525373 for Android raptor tests.

Bug 1525373 had been waiting for a while for aarch64 windows fixes that
are still not there, and landed with a workaround for those. But while
waiting, bug 1555479 changed the run-task transform, making bug 1525373
double-wrap bitbar commands with /builds/taskcluster/script.py.

So remove the now redundant wrapping.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7fafd56a85de
Support comm-checkout when using generic-worker with run-task. r=glandium

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e7fc0aba600f
Fixup bug 1525373 for Android raptor tests. r=tomprince

Comment 28

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/7fafd56a85de
https://hg.mozilla.org/mozilla-central/rev/e7fc0aba600f