Closed Bug 1525373 Opened 5 years ago Closed 5 years ago

Use run-task wrapper everywhere

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set
normal

Tracking

(firefox70 fixed)

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: catlee, Assigned: glandium)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(6 files, 2 obsolete files)

At the moment, Windows and macOS tests aren't using run-task, which means they don't get nice timestamped logging.

I think we should try and make sure all tasks run under run-task.

Blocks: 1527066
Depends on: 1436037

In bug 1534463 I patched run-task to enable Mercurial operations to run against a private web head dedicated to CI. As part of testing that change on try, I discovered this bug. I might end up picking this up in the next little while, so we have more tasks performing checkouts against the private service, and so I don't need to re-implement that logic in various places.

Blocks: hgwebaws
Assignee: nobody → mh+mozilla
Blocks: 1525218, 1543624
Depends on: 1543915
Depends on: 1544983
No longer depends on: 1544983
Depends on: 1544986
Depends on: 1534578
Depends on: 1545339
Depends on: 1545343
Depends on: 1545344

The generic-worker tasks on bitbar, for some reason, don't have a PATH
set at all, and have it set via /builds/taskcluster/script.py. But the
command that sets run-task up runs outside that, so we have to be
explicit about where to find chmod.

The macos workers have two python 2.7 installed: one in /usr/bin, and
one in /usr/local/bin. For some reason, the one in /usr/local/bin is
broken wrt SSL.

When running the current mozharness command directly without a full path
to python2.7, generic-worker executes it using its own PATH, and that
uses /usr/bin/python2.7. When wrapping with something else, though
(run-task, but that would apply just as much with sh -c), the PATH
from the task itself is used, and it's explicitly set to use
/usr/local/bin first, and the broken python 2.7 then gets used.

We could change the PATH, but there might be other things relying on the
current order, so just use /usr/bin/python2.7.

Depends on D28023

The tasks on bitbar currently rely on being run as root, and run-task
defaults to drop its privileges to the worker user.

Depends on D28024

mozharness and mozharness test transforms currently do their own version
of wrapping commands with run-task in docker-worker. This makes them
piggy back on the run_task transform instead.

Some things then become redundant with what the run_task transform does,
and some others need to happen later than they currently do in order to
work.

This has the side effect of enabling some of the more recent run-task
features, like --fetch-hgfingerprint.

Depends on D28025

mozharness and mozharness test transforms for generic-worker currently
don't wrap the commands with run-task. This changes things such that the
commands are wrapped with run-task, by piggy-backing on the run_task
transform.

Some things then become redundant with what the run_task transform does,
and some others need to happen later than they currently do in order to
work.

Depends on D28026

Depends on: 1546485
Attachment #9059197 - Attachment is obsolete: true
Depends on: 1551789
Attached file Bug 1525373 - Work around bug 1551789. (obsolete) —

TASKCLUSTER_ROOT_URL is not set on gecko-t-win10-64-hw workers, so set it
via transforms.

Attachment #9065318 - Attachment is obsolete: true
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/1fa5952696ca
Use an explicit path for python2.7 for mozharness. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/37f758cfb92a
Add an option to `using: run-task` to keep running tasks as root. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/00c0dfc00b1b
Use run_task transform for mozharness on docker-worker. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/108a519aff66
Use run-task for mozharness jobs on generic-worker. r=tomprince
Flags: needinfo?(mh+mozilla)
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/3e066ee9fbca
Use an explicit path for python2.7 for mozharness. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/58087407b5b9
Add an option to `using: run-task` to keep running tasks as root. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/447b8da8ebb3
Use run_task transform for mozharness on docker-worker. r=tomprince
https://hg.mozilla.org/integration/autoland/rev/43cece5c03b4
Use run-task for mozharness jobs on generic-worker. r=tomprince
Depends on: 1557614

Backed out changeset 43cece5c03b4 (Bug 1525373) for Windows 10 AArch failures

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=windows%2C10%2Caarch64%2Copt%2Ctest-windows10-aarch64%2Fopt-cppunit-1proc%2C%28cpp%29&fromchange=fb6f02f359d7883555d3c3590f18f71f484aa16e&tochange=9183f80b6bf0d03510336ac8faa0b98f50b56e9f&selectedJob=250538476

Backout link: https://hg.mozilla.org/integration/autoland/rev/9183f80b6bf0d03510336ac8faa0b98f50b56e9f

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250538476&repo=autoland&lineNumber=40

[taskcluster 2019-06-07T03:43:54.496Z] Task ID: PX9Vk69hSfOYv4GF4SOJUg
[taskcluster 2019-06-07T03:43:54.496Z] === Task Starting ===
[taskcluster 2019-06-07T03:43:55.447Z] Uploading redirect artifact public/logs/live.log to URL http://1.1.1.1:49940/log/C3fiq8_oR1aR42KfAjzYcQ with mime type "text/plain; charset=utf-8" and expiry 2019-06-07T04:58:55.029Z
[taskcluster 2019-06-07T03:43:55.821Z] [mounts] Downloading task PmX1rNt0S7WJbay2Gv2WEQ artifact public/build/mozharness.zip to downloads\JeX8zpXwRGG80dm4pOWWcw
[taskcluster 2019-06-07T03:43:56.495Z] [mounts] Downloaded 2342280 bytes with SHA256 2a766b4de461228351527418f0801051f72d83e58c5ab2811b4922b0d3a3540e from task PmX1rNt0S7WJbay2Gv2WEQ artifact public/build/mozharness.zip to downloads\JeX8zpXwRGG80dm4pOWWcw
[taskcluster:warn 2019-06-07T03:43:56.496Z] [mounts] Download downloads\JeX8zpXwRGG80dm4pOWWcw of task PmX1rNt0S7WJbay2Gv2WEQ artifact public/build/mozharness.zip has SHA256 2a766b4de461228351527418f0801051f72d83e58c5ab2811b4922b0d3a3540e but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2019-06-07T03:43:56.496Z] [mounts] Creating directory C:\tasks\task_1559877973 with permissions 0700
[taskcluster 2019-06-07T03:43:56.496Z] [mounts] Extracting zip file downloads\JeX8zpXwRGG80dm4pOWWcw to 'C:\tasks\task_1559877973'
[taskcluster 2019-06-07T03:43:57.012Z] [mounts] Downloading url https://hg.mozilla.org/integration/autoland/raw-file/43cece5c03b4ff1d1400a5be6b5fa255e8fcd8bf/taskcluster/scripts/run-task to downloads\Njc52SgMTwSjMtJ8zWrqsw
[taskcluster 2019-06-07T03:43:57.709Z] [mounts] Downloaded 30921 bytes with SHA256 8be03508dc6d71e4397d7d13eafb8927ebc47c36d5420fd12034c72fc6a50d30 from url https://hg.mozilla.org/integration/autoland/raw-file/43cece5c03b4ff1d1400a5be6b5fa255e8fcd8bf/taskcluster/scripts/run-task to downloads\Njc52SgMTwSjMtJ8zWrqsw
[taskcluster:warn 2019-06-07T03:43:57.709Z] [mounts] Download downloads\Njc52SgMTwSjMtJ8zWrqsw of url https://hg.mozilla.org/integration/autoland/raw-file/43cece5c03b4ff1d1400a5be6b5fa255e8fcd8bf/taskcluster/scripts/run-task has SHA256 8be03508dc6d71e4397d7d13eafb8927ebc47c36d5420fd12034c72fc6a50d30 but task payload does not declare a required value, so content authenticity cannot be verified
[taskcluster 2019-06-07T03:43:57.709Z] [mounts] Creating directory C:\tasks\task_1559877973 with permissions 0700
[taskcluster 2019-06-07T03:43:57.710Z] [mounts] Copying downloads\Njc52SgMTwSjMtJ8zWrqsw to C:\tasks\task_1559877973\run-task
[taskcluster 2019-06-07T03:43:58.029Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --cppunittest-suite=cppunittest --disable-e10s --download-symbols ondemand --cppunittest-suite=cppunittest --disable-e10s
C:\tasks\task_1559877973>C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --cppunittest-suite=cppunittest --disable-e10s --download-symbols ondemand --cppunittest-suite=cppunittest --disable-e10s
[taskcluster 2019-06-07T03:43:58.334Z] Exit Code: -1073741515
[taskcluster 2019-06-07T03:43:58.334Z] User Time: 15.625ms
[taskcluster 2019-06-07T03:43:58.334Z] Kernel Time: 78.125ms
[taskcluster 2019-06-07T03:43:58.334Z] Wall Time: 298.0019ms
[taskcluster 2019-06-07T03:43:58.334Z] Result: FAILED
[taskcluster 2019-06-07T03:43:58.334Z] === Task Finished ===
[taskcluster 2019-06-07T03:43:58.334Z] Task Duration: 305.0053ms
[taskcluster:error] Uploading error artifact public/logs from file logs with message "Could not read directory 'C:\tasks\task_1559877973\logs'", reason "file-missing-on-worker" and expiry 2020-06-06T03:30:42.803Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\logs'
[taskcluster:error] Uploading error artifact public/test_info from file build/blobber_upload_dir with message "Could not read directory 'C:\tasks\task_1559877973\build\blobber_upload_dir'", reason "file-missing-on-worker" and expiry 2020-06-06T03:30:42.803Z
[taskcluster:error] TASK FAILURE during artifact upload: file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\build\blobber_upload_dir'
[taskcluster 2019-06-07T03:43:59.645Z] Uploading redirect artifact public/logs/live.log to URL https://queue.taskcluster.net/v1/task/PX9Vk69hSfOYv4GF4SOJUg/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2020-06-06T03:30:42.803Z
[taskcluster:error] exit status -1073741515
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\logs'
[taskcluster:error] file-missing-on-worker: Could not read directory 'C:\tasks\task_1559877973\build\blobber_upload_dir'

No longer depends on: 1557614
Flags: needinfo?(mh+mozilla)

Bug 1557614 is what caused those problems.

Flags: needinfo?(mh+mozilla)
Keywords: leave-open
See Also: → 1557764
Keywords: leave-open
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/4d0c22c47c7f
Use run-task for mozharness jobs on generic-worker. r=tomprince
Blocks: 1568349
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/integration/autoland/rev/ba6240f6bc90
Support comm-checkout when using generic-worker with run-task. r=glandium
Regressed by: 1569091
Keywords: regression
Regressions: 1569459

Filed bug 1569459 to track those raptor failures.

Bug 1525373 had been waiting for a while for aarch64 windows fixes that
are still not there, and landed with a workaround for those. But while
waiting, bug 1555479 changed the run-task transform, making bug 1525373
double-wrap bitbar commands with /builds/taskcluster/script.py.

So remove the now redundant wrapping.

Pushed by csabou@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7fafd56a85de
Support comm-checkout when using generic-worker with run-task. r=glandium
Flags: needinfo?(mh+mozilla)
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/e7fc0aba600f
Fixup bug 1525373 for Android raptor tests. r=tomprince
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: