1847288 - mach bootstrap / build for release source, follow up from Bug 1798746

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, task)

Description

[Nicklas Boman [:smurfd]]

This is a follow up bug from :
Bug 1798746 - Ensure checking for mercurial/git only if its a hg/git clone, not release source

It landed, and then it turned out that everything in third_party/python/taskcluster_taskgraph/taskgraph/ gets overwritten when Taskgraph gets updated. So the changes were reverted.

I created a https://github.com/taskcluster/taskgraph/pull/291 (which we closed today)
@ahal and me had some chatting, he expressed interest in not touching taskgraph code that much for this special Firefox case. Which makes all the sense.

And im thinking i would, just above https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/python/mozbuild/mozbuild/toolchains.py#20
call get_repository() in a try/except RuntimeError. If it hits RuntimeError i would check if a moz.configure file exists. Then make whatever calls i need from above callstack from the taskcluster things, either by hand or calling the actual functions from python/mozbuild/mozbuild/toolchains.py

If we dont hit the exception, we continue as usual.

This is the callstack for this.

maybe_install_private_packages_or_exit() is called from bootstrap() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/python/mozboot/mozboot/bootstrap.py#348
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/python/mozboot/mozboot/bootstrap.py#394

install_toolchain_artifact() is called from maybe_install_private_packages_or_exit() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/python/mozboot/mozboot/bootstrap.py#263

bootstrap_toolchain() is called from install_toolchain_artifact() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/python/mozboot/mozboot/base.py#347

bootstrap_path() is called from bootstrap_toolchain() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/python/mozbuild/mozbuild/bootstrap.py#52

bootstrap_toolchain_tasks() is called from bootstrap_path() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/build/moz.configure/bootstrap.configure#151

toolchain_task_definitions() is called from bootstrap_toolchain_tasks() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/build/moz.configure/bootstrap.configure#110

load_tasks_for_kind() is called from toolchain_task_definitions() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/python/mozbuild/mozbuild/toolchains.py#20

parameters_loader() is called from load_tasks_for_kind() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/generator.py#444

load_parameters_file() is called from parameters_loader() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/parameters.py#359

Parameters.init() is called from load_parameters_file() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/parameters.py#310

fill_defaults() is called from Parameters.init() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/parameters.py#159

get_defaults() via defaults_functions is called from fill_defaults() or extend_parameters_schema in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/parameters.py#146
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/parameters.py#198

get_repository() is called from get_defaults() in:
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/parameters.py#82

get_repository()
https://searchfox.org/mozilla-central/rev/ddff4872b8220fae71155e39fd30ac3b760a4741/third_party/python/taskcluster_taskgraph/taskgraph/util/vcs.py#505

Comment 1

[betterbird.project+8]

Attachment: 5_build.txt

We're a Thunderbird downstream project trying to build a modified Thunderbird via tarball in GitHub actions. Sadly on Mac that doesn't work, that is mach bootstrap fails.
https://github.com/Betterbird/thunderbird-patches/issues/175
https://github.com/sagudev/thunderbird-patches/actions/runs/5534808044/jobs/10100251329#step:5:535

We don't know for how long GitHub actions keeps the logs, so we'll attach the relevant file. Building from tarball on Linux is fine. We much prefer to build from tarball to save having to pull the 2GB+ repo each time, the tarball is "only" 500 MB.

Sorry for the intrusion, we just saw some more action on the issue. If there is interest, we can file another bug.

Comment 2

[Nicklas Boman [:smurfd]]

Thanks for the info. Maby you can try disabling sccache on mac?

Comment 3

[Nicklas Boman [:smurfd]]

Attachment: Bug 1847288 - mach bootstrap / build for release source

Comment 4

[Nicklas Boman [:smurfd]]

^^ major wip (not close to working yet) I will need info when i ...
plan is to get it working then remove stuff not needed. for it to not be a massive patch.

Comment 5

[Nicklas Boman [:smurfd]]

Hey @ahal,
when time, could you please check the phabricator. have some questions.
hoping also to spark some idea by typing something down, if so i cancel the need info :)

Comment 6

[Nicklas Boman [:smurfd]]

Update,
I have things working now (or atleast it downloads the toolchains artifacts, clang, clang-tidy, minidump-stackwalk and so on)
Have not updated the phabricator revision yet. will see if it builds first.

Comment 7

[Nicklas Boman [:smurfd]]

hey @ahal
more updates
this works now, have also built successfully on my mac.
Updated the phabricator revision.

Keeping it wip for now

Comment 8

[Andrew Halberstadt [:ahal]]

Thanks, I commented on the patch. I don't think we should be duplicating TaskGraphGenerator or Parameters classes here. I was hoping all that would be needed is some light monkeypatching. I'm a bit confused because the changes in your PR to upstream Taskgraph seemed to be fairly trivial, but now it seems much more complicated.

Comment 9

[Nicklas Boman [:smurfd]]

That sounds Great!, because yes i was not happy to have to copy most of the classes and only having modified one function in it.
im not sure ive done this in python before, but i get the idea.
In the Parameters class for example, i have only touched the file_url(self, path, pretty=False) function

i would do something like this?!

from taskgraph.parameters import Parameters
def parameters_file_url(self, path, pretty=False): # my modified function
    .....

def load_parameters_file(....):
    ....
    if not spec:
        taskgraph.parameters.Parameters.file_url = parameters_file_url
        return Parameters(strict=strict, repo_root=repo_root, **overrides)

---- edit ----
nevermind, i think you ment to recreate the PR to taskgraph into monkey patched functions here.
will look more into that and get back with a Needinfo when i have something.

Comment 10

[Nicklas Boman [:smurfd]]

Never mind that, just got it working.
submitted updated revision to phabricator

Comment 11

[Nicklas Boman [:smurfd]]

Now im pretty sure we are at a minimum patch for this :)

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: GitHub Pull Request

Hi Nicklas, could you try applying this patch to the in-tree copy of Taskgraph and letting me know if it's sufficient to unblock you? Thanks!

Comment 13

[Nicklas Boman [:smurfd]]

Hey Andrew, Yes! that works, it will download the artifacts even without .hg or .git folder :)
Thanks!!
I guess we will abandon the phabricator revision, then when that patch gets bumped into third_party/ we close this bug?

Comment 14

[Andrew Halberstadt [:ahal]]

Thanks for checking!

Yep, that would be the plan. I'll get this PR reviewed and landed, then make a new Taskgraph release so we can pick it up in Gecko.

Comment 15

[Andrew Halberstadt [:ahal]]

The change should now be in mozilla-central so this should be unblocked again. Sorry for the delay!

Comment 16

[Mike Hommey [:glandium]]

Closing per comment 15.