1798746 - Ensure checking for mercurial/git only if its a hg/git clone, not release source

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect, P5)

Description

[Nicklas Boman [:smurfd]]

After receiving
https://bugzilla.mozilla.org/show_bug.cgi?id=1780449#c4
I realize that we should only check for mercurial/git if it is a mercurial or git checkout.

So move these two :
https://searchfox.org/mozilla-central/rev/aa1bb4f0ca8bfda4b117d1befca47b72d5dd6d5d/python/mozboot/mozboot/bootstrap.py#327-328

to like this row:
https://searchfox.org/mozilla-central/rev/aa1bb4f0ca8bfda4b117d1befca47b72d5dd6d5d/python/mozboot/mozboot/bootstrap.py#339
and add a :
if checkout_type == "hg": above that

This makes sense, right?

Comment 1

[Nicklas Boman [:smurfd]]

Hey ahochheiden,
please forward need info if someone is more suited.

But i just want to reason about this, since it was not such an easy slam-dunk as i thought above. And im not even sure it has todo with my patch.
In https://bugzilla.mozilla.org/show_bug.cgi?id=1780449#c4 b5 makes a valid point though.

If you have a release-source, you should be able to build it with ./mach bootstrap && ./mach build right?
Those will never be a git or mercurial checkout. Tried to revert the patch but i do still get that issue, it raises bootstrap.py#622-625

Should we maby just change the text :
https://searchfox.org/mozilla-central/rev/aa1bb4f0ca8bfda4b117d1befca47b72d5dd6d5d/python/mozboot/mozboot/bootstrap.py#622-625
to say something like "or is this a release source, follow this guildeline https://some.link/path/to/build-source-guide/"

thanks in advance!

Comment 2

[Mike Hommey [:glandium]]

mach bootstrap is currently not supported from a source tarball. The code you pointed at wouldn't be enough to make it work, although your suggested change probably does make sense in the sense that we don't need mercurial when the user is using git.

Comment 3

[b5]

Thanks for looking into the suggestion of making mach bootstrap work on a source tarball. The suggestion in comment #0 to move
https://searchfox.org/mozilla-central/rev/aa1bb4f0ca8bfda4b117d1befca47b72d5dd6d5d/python/mozboot/mozboot/bootstrap.py#327-328
doesn't work, since the variable hg is used at line #334.

Instead, as a first step
https://searchfox.org/mozilla-central/rev/aa1bb4f0ca8bfda4b117d1befca47b72d5dd6d5d/python/mozboot/mozboot/bootstrap.py#622-625
needs to be replaced with return "none", path and the while loop at line #589 needs to be removed to avoid climbing up the directory tree. With those changes mach bootstrap works for an artifact build (option 1), but not for a "real" build (option 2). It does in fact install a bunch of packages but then stops here:
https://searchfox.org/mozilla-central/rev/aa1bb4f0ca8bfda4b117d1befca47b72d5dd6d5d/third_party/python/taskcluster_taskgraph/taskgraph/util/vcs.py#508
without installing the "harder" stuff like node, llvm/clang.

As comment #2 said, there's more work in it.

Comment 4

[Nicklas Boman [:smurfd]]

(In reply to Mike Hommey [:glandium] from comment #2)

mach bootstrap is currently not supported from a source tarball. The code you pointed at wouldn't be enough to make it work, although your suggested change probably does make sense in the sense that we don't need mercurial when the user is using git.

Right, you build release source like :
tar Jxf firefox-106.0.source.tar.xz && cd firefox-106.0 && mkdir obj && cd obj && ../configure && make && make install

There must be alot of reasons why mach bootstrap has not been implemented for release-source.
I dont feel suited to answer what todo next.

Comment 5

[Nicklas Boman [:smurfd]]

i might have an idea, will try on a clean install tomorrow
after this, add a line :
https://searchfox.org/mozilla-central/rev/49011d374b626d5f0e7dc751a8a57365878e65f1/python/mozboot/mozboot/bootstrap.py#589-591
moz_configure = path / "moz.configure"
remove https://searchfox.org/mozilla-central/rev/49011d374b626d5f0e7dc751a8a57365878e65f1/python/mozboot/mozboot/bootstrap.py#613
After this https://searchfox.org/mozilla-central/rev/49011d374b626d5f0e7dc751a8a57365878e65f1/python/mozboot/mozboot/bootstrap.py#612-616
add :

elif moz_configure.exists():
    return "release", path

Comment 6

[b5]

(In reply to Nicklas Boman [:smurfd] from comment #4)

Right, you build release source like :
tar Jxf firefox-106.0.source.tar.xz && cd firefox-106.0 && mkdir obj && cd obj && ../configure && make && make install

The problem is that you need to set up the environment (node, llvm/clang, cbindgen, etc.) first for configure to succeed.

Please note comment #3: After overcoming the issues in bootstrap.py, more issues arose in third_party/python/taskcluster_taskgraph/taskgraph/util/vcs.py.

Comment 7

[Alex Hochheiden [:ahochheiden] [he/him]]

You are welcome to work on this and I will review the patches you submit, but adding support for ./mach bootstrap from the source tarball is not a priority for the build team at this time.

Comment 8

[Nicklas Boman [:smurfd]]

Attachment: Bug 1798746 - Ensure checking for mercurial/git only if its a hg/git clone, not release source r=ahochheiden

Comment 9

[Nicklas Boman [:smurfd]]

the patch is not working, and is messy and filled with print statements (hence the WIP)
someone happen to notice and see something obvious, im very open for suggestions :)
like if this is the right way or if can shortcut at some point to ot have to follow the whole way as a git or hg repo does.

Comment 10

[Nicklas Boman [:smurfd]]

some progress, now it downloads the toolchains, cbindgen, minidump-stackwalk, nasm, clang, sccache, sysroot-wasm32-wasi, pkgconfig, sysroot-x86_64-linux-gnu, fix-stackwalk & dump_syms
then unfortunately crashes saying : Exception: python/mozboot/**/android did not match anything
but it has already found that atleast one time....

Comment 11

[Nicklas Boman [:smurfd]]

works great on mac now, bootstraps and builds.
linux bootstrap fails for clang-tidy download, not sure why, but seems to build...

Comment 12

[Nicklas Boman [:smurfd]]

what is strange is that it ends like this on linux :

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/home/casper/.mozbuild/srcdirs/mc-554bee543f1f/_virtualenvs/mach/bin/python', '/home/casper/code/mc/mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'linux64-clang-tidy']' returned non-zero exit status 1.

  File "/home/casper/code/mc/python/mozboot/mozboot/mach_commands.py", line 41, in bootstrap
    bootstrapper.bootstrap(command_context.settings)
  File "/home/casper/code/mc/python/mozboot/mozboot/bootstrap.py", line 384, in bootstrap
    self.maybe_install_private_packages_or_exit(application)
  File "/home/casper/code/mc/python/mozboot/mozboot/bootstrap.py", line 256, in maybe_install_private_packages_or_exit
    self.instance.ensure_clang_static_analysis_package()
  File "/home/casper/code/mc/python/mozboot/mozboot/linux_common.py", line 30, in ensure_clang_static_analysis_package
    self.install_toolchain_static_analysis(static_analysis.LINUX_CLANG_TIDY)
  File "/home/casper/code/mc/python/mozboot/mozboot/base.py", line 355, in install_toolchain_static_analysis
    self.install_toolchain_artifact_impl(clang_tools_path, toolchain_job)
  File "/home/casper/code/mc/python/mozboot/mozboot/base.py", line 395, in install_toolchain_artifact_impl
    subprocess.check_call(cmd, cwd=str(install_dir))
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)

but if i run this command from the same shell as i tried to bootstrap from:
/home/casper/.mozbuild/srcdirs/mc-554bee543f1f/_virtualenvs/mach/bin/python /home/casper/code/mc/mach artifact toolchain --bootstrap --from-build linux64-clang-tidy
it downloads clang tidy.
dont get it :|

---- edit
and if i set install_dir to "." it downloads and installs though in the m-c root, not the ~/.mozbuild/clang-tools dir
https://searchfox.org/mozilla-central/rev/9b8ebf06145feeccf34dc126cf45b07e86556392/python/mozboot/mozboot/base.py#391
hmm whats the difference of clang-tidy and clang...

Comment 13

[Nicklas Boman [:smurfd]]

@ahochheiden, hey, i would say im very close(i think :)) to having a working patch. Will add more comments and rewrite some parts that can be written better.
It downloads all artifacts and untars them without a problem, accept the clang-tidy one, on linux.
if i would replace this line
https://searchfox.org/mozilla-central/rev/c382c6a7ca17e86060f353d8f580f1df47c1442a/python/mozboot/mozboot/base.py#391
with :

if toolchain_job is "linux64-clang-tidy":
    bootstrap_toolchain(toolchain_job)
else:
    subprocess.check_call(cmd, cwd=str(install_dir))

It would run through the bootstrap process without a problem, but untar clang-tidy in ~/.mozboot instead cof ~/.mozboot/clang-tools
And as i wrote above, if i run the actual commands from shell, with my patch applied, it will download clang-tidy but untars it in mozilla-central folder instead of ~/.mozboot
/home/casper/.mozbuild/srcdirs/mc-554bee543f1f/_virtualenvs/mach/bin/python /home/casper/code/mc/mach artifact toolchain --bootstrap --from-build linux64-clang-tidy

What im trying to ask is, why is clang-tidy handled special from say clang, and kindof what is the difference from Mac and Linux's handling of the clang-tidy artifact...
and also tips and ideas where i might have gone wrong :)
thanks in advance.

Comment 14

[Nicklas Boman [:smurfd]]

the problem has todo with the digest hash for clang-tidy does not get the correct value

Comment 15

[Nicklas Boman [:smurfd]]

Hey again, realized that clang-tidy is probably not important for a source checkout
So we ignore that for now.
please have a look when time is available.
im very open to changes :)

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The following field has been copied from a duplicate bug:

Field	Value	Source
Regressed by	bug 1662893	bug 1687908

For more information, please visit auto_nag documentation.

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

This bug has the keyword regression, so its type should be defect.

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1662893

Comment 20

[Nicklas Boman [:smurfd]]

rebased and fixed some linting

Comment 21

[Alex Hochheiden [:ahochheiden] [he/him]]

Reviewed.

Comment 22

[Nicklas Boman [:smurfd]]

Hey @ahochheiden, thanks for asking me to double check.
It now works both for hg, git and source.

I had 3 issues, two minor and 1 big.

First issue was that in python/mozboot/mozboot/bootstrap.py there

if checkout_type != "SOURCE":
    self.instance.ensure_clang_static_analysis_package()

could be removed since ensure_clang_static_analysis_package() is not used anymore

In third_party/python/taskcluster_taskgraph/taskgraph/parameters.py i had indenting issues, using only 2 spaces instead of 4
In there i had also managed to add the else statement inside the above return statement, causing it to not work at all, for anything... (was also quite tricky to find ;))

so yes please, land when possible :)

Comment 23

[Alex Hochheiden [:ahochheiden] [he/him]]

Thanks! Landing is queued.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/02cc74634f7d
Ensure checking for mercurial/git only if its a hg/git clone, not release source r=ahochheiden

Comment 25

[Iulian Moraru]

Backed out for causing build bustages.

• Push with failures

• Failure log

• Backout link

[task 2023-05-03T16:29:22.690Z] make[1]: Entering directory '/builds/worker/workspace/sm-package/mozjs-114.0a1.0/obj-spider/js/src'
[task 2023-05-03T16:29:22.690Z] ../../dist/bin/run-mozilla.sh /builds/worker/workspace/sm-package/mozjs-114.0a1.0/obj-spider/_virtualenvs/build/bin/python -u /builds/worker/workspace/sm-package/mozjs-114.0a1.0/js/src/tests/jstests.py \
[task 2023-05-03T16:29:22.690Z] 	--no-progress --format=automation --timeout 300 \
[task 2023-05-03T16:29:22.690Z] 	--args='--dll /builds/worker/fetches/injector/libbreakpadinjector.so' --jitflags=jstests \
[task 2023-05-03T16:29:22.690Z] 	../../dist/bin/js
[task 2023-05-03T16:29:23.237Z] Traceback (most recent call last):
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/js/src/tests/jstests.py", line 874, in <module>
[task 2023-05-03T16:29:23.237Z]     sys.exit(main())
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/js/src/tests/jstests.py", line 756, in main
[task 2023-05-03T16:29:23.237Z]     test_count, test_gen = load_tests(options, requested_paths, excluded_paths)
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/js/src/tests/jstests.py", line 671, in load_tests
[task 2023-05-03T16:29:23.237Z]     wpt_tests = load_wpt_tests(xul_tester, requested_paths, excluded_paths)
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/js/src/tests/jstests.py", line 552, in load_wpt_tests
[task 2023-05-03T16:29:23.237Z]     test_manifests = manifestupdate.run(
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/testing/web-platform/manifestupdate.py", line 136, in run
[task 2023-05-03T16:29:23.237Z]     manifestdownload.download_from_taskcluster(
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/testing/web-platform/manifestdownload.py", line 219, in download_from_taskcluster
[task 2023-05-03T16:29:23.237Z]     return download_manifest(
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/testing/web-platform/manifestdownload.py", line 166, in download_manifest
[task 2023-05-03T16:29:23.237Z]     commits = commits_func()
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/testing/web-platform/manifestdownload.py", line 222, in <lambda>
[task 2023-05-03T16:29:23.237Z]     lambda: get_commits(logger, repo_root),
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/testing/web-platform/manifestdownload.py", line 61, in get_commits
[task 2023-05-03T16:29:23.237Z]     changeset_iter = repo._run(
[task 2023-05-03T16:29:23.237Z]   File "/builds/worker/workspace/sm-package/mozjs-114.0a1.0/python/mozversioncontrol/mozversioncontrol/__init__.py", line 99, in _run
[task 2023-05-03T16:29:23.238Z]     return subprocess.check_output(
[task 2023-05-03T16:29:23.238Z]   File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
[task 2023-05-03T16:29:23.238Z]     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
[task 2023-05-03T16:29:23.238Z]   File "/usr/lib/python3.9/subprocess.py", line 505, in run
[task 2023-05-03T16:29:23.238Z]     with Popen(*popenargs, **kwargs) as process:
[task 2023-05-03T16:29:23.238Z]   File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
[task 2023-05-03T16:29:23.239Z]     self._execute_child(args, executable, preexec_fn, close_fds,
[task 2023-05-03T16:29:23.239Z]   File "/usr/lib/python3.9/subprocess.py", line 1756, in _execute_child
[task 2023-05-03T16:29:23.239Z]     self.pid = _posixsubprocess.fork_exec(
[task 2023-05-03T16:29:23.239Z] TypeError: expected str, bytes or os.PathLike object, not NoneType
[task 2023-05-03T16:29:23.271Z] make[1]: *** [Makefile:63: check-jstests] Error 1
[task 2023-05-03T16:29:23.271Z] make[1]: Leaving directory '/builds/worker/workspace/sm-package/mozjs-114.0a1.0/obj-spider/js/src'
[task 2023-05-03T16:29:23.271Z] make: *** [Makefile:220: check-jstests] Error 2
[task 2023-05-03T16:29:23.272Z] in directory /builds/worker/workspace/sm-package/mozjs-114.0a1.0/obj-spider, running ['/builds/worker/workspace/sm-package/mozjs-114.0a1.0/obj-spider/_virtualenvs/build/bin/python3', '/builds/worker/workspace/sm-package/mozjs-114.0a1.0/testing/mozbase/mozcrash/mozcrash/mozcrash.py', '/tmp', '/builds/worker/workspace/sm-package/mozjs-114.0a1.0/obj-spider/dist/crashreporter-symbols']
[task 2023-05-03T16:29:23.362Z] mozcrash checking /tmp for minidumps...
[task 2023-05-03T16:29:23.373Z] exit status 0 for '(make-nonempty)'
[task 2023-05-03T16:29:23.373Z] exit status 0 for 'make check-jit-test'
[task 2023-05-03T16:29:23.373Z] exit status 0 for 'jsapi-tests'
[task 2023-05-03T16:29:23.373Z] exit status 2 for 'jstests'
[task 2023-05-03T16:29:23.381Z] + status=2
[task 2023-05-03T16:29:23.381Z] + cp -rL ./mozjs-114.0a1.0/obj-spider/dist/bin/js ./mozjs-114.0a1.0/obj-spider/dist/bin/jsapi-tests ./mozjs-114.0a1.0/obj-spider/dist/bin/js-gdb.py ./mozjs-114.0a1.0/obj-spider/dist/bin/libmozjs-114a1.so /builds/worker/artifacts
[task 2023-05-03T16:29:24.210Z] + exit 2
[taskcluster 2023-05-03 16:29:28.530Z] === Task Finished ===
[taskcluster 2023-05-03 16:30:39.064Z] Unsuccessful task run with exit code: 2 completed in 555.942 seconds

Comment 26

[Norisz Fay [:noriszfay]]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/f2daa00570fe

Comment 27

[Nicklas Boman [:smurfd]]

thanks for backing it out, ill try to reproduce it locally

Comment 28

[Nicklas Boman [:smurfd]]

So i might have an idea.
It seems to error when when running tests for Spidermonkey or the package for spidermonkey.

[task 2023-05-03T16:29:23.239Z] TypeError: expected str, bytes or os.PathLike object, not NoneType
[task 2023-05-03T16:29:23.271Z] make[1]: *** [Makefile:63: check-jstests] Error 1
[task 2023-05-03T16:29:23.271Z] make[1]: Leaving directory '/builds/worker/workspace/sm-package/mozjs-114.0a1.0/obj-spider/js/src'
[task 2023-05-03T16:29:23.271Z] make: *** [Makefile:220: check-jstests] Error 2

I notice that the latest release of mozjs does not have a moz.configure file in the tar file, but a moz.build file
so im guessing that it will not classify as either a hg, git or source repository.

Would it make sense to check for the moz.build file instead to ensure its a source folder?
How would one reproduce the problem from a hg source?
I though and tried to do: ./mach bootstrap and chose 5 for spidermonkey build
move away the .hg folder and the moz.configure file
´./mach buildand then./mach jsapi-testsbut that does not seem to work for me. hangs ontestStencil_OffThreadDecodePinned` both with and without the patch.

Comment 29

[Alex Hochheiden [:ahochheiden] [he/him]]

Do we need to check for anything at all? What's the reason for not assuming a SOURCE repo if neither a .hg and .git folder are present?

Comment 30

[Alex Hochheiden [:ahochheiden] [he/him]]

Ah, I looked at the code. It's to determine if we're in the root directory (and need to stop going up). I think moz.build would be a bad file to check for in that case, since there are plenty of duplicates of those in the tree. How about the mach file? That should be unique, and always in the root directory, since if it's not there you can't use any commands.

Comment 31

[Nicklas Boman [:smurfd]]

I was later yesterday able to actually reproduce the error. makes things way easier to understand, and not to just guess (especially since i guessed wrong :)) what the issue is :)

reproduced it by :
bootstrapping mozilla-central with option 5 for js build
building mozilla-central
in js/src i ran ./make-source-package.py
made a directory outside of mozilla-central
untared /tmp/mozjs-.xz there
entered the untarred mozjs folder.
./mach build
./mach jstests

the actual issue is this part :
https://searchfox.org/mozilla-central/rev/77cadb613712e36bb4bde402fc5e9d645b3565f5/python/mozversioncontrol/mozversioncontrol/__init__.py#97-105

If self._tool is None
return subprocess.check_output returns None which causes issues

        cmd = (str(self._tool),) + args
        try:
            return subprocess.check_output(
                cmd, cwd=self.path, env=self._env, universal_newlines=True
            )
        except subprocess.CalledProcessError as e:
            if e.returncode in return_codes:
                return ""
            raise

If i add the try statement in a if self._tool return "str" else try it launches the jstests. will verify that the bootstrapping works aswell

        cmd = (str(self._tool),) + args
        if not self._tool: 
            return "str"
        else:
            try:
                return subprocess.check_output(
                    cmd, cwd=self.path, env=self._env, universal_newlines=True
                )
            except subprocess.CalledProcessError as e:
                if e.returncode in return_codes:
                    return ""
                raise

Comment 32

[Nicklas Boman [:smurfd]]

submitted my idea for review, let me know what you think :)
tried both hg, git and source bootstrap + the failing jstests

Comment 33

[Alex Hochheiden [:ahochheiden] [he/him]]

(In reply to Nicklas Boman [:smurfd] from comment #32)

submitted my idea for review, let me know what you think :)
tried both hg, git and source bootstrap + the failing jstests

Sounds good to me.

Comment 34

[Nicklas Boman [:smurfd]]

thanks, yes now it looks alot better.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fab57c6553f4
Ensure checking for mercurial/git only if its a hg/git clone, not release source r=ahochheiden

Comment 36

[Cosmin Sabou [:CosminS]]

Backed out for causing win aarch64 build bustages.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=windows%2C2012%2Caarch64%2Copt%2Cbuild-win64-aarch64-eme%2Fopt%2Cbe&fromchange=fab57c6553f439f5f4aed0c6f0cf0a8d52fb1575&tochange=048e66b392d693276850f2fc2cb61c40a6aa95e6&selectedTaskRun=Xx_dmDxPSECj8lFGfYEVQg.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=415030188&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/048e66b392d693276850f2fc2cb61c40a6aa95e6

Comment 37

[Nicklas Boman [:smurfd]]

Thanks for the backout!
3rd time is the charm crosses fingers

https://searchfox.org/mozilla-central/rev/373d05f4eabdb90a6480d5d36d983b8bff36c9d8/python/mozbuild/mozbuild/artifact_commands.py#186
I had a check here before the return, that if source=None and distdir=None return install_from() with a argument src=True
i believe it was leftovers from early attempts.
unfortunately it was not hit during local tests.
sorry about this.

Comment 38

[Alex Hochheiden [:ahochheiden] [he/him]]

Pushed.

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ebd8e5e25327
Ensure checking for mercurial/git only if its a hg/git clone, not release source r=ahochheiden

Comment 40

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/ebd8e5e25327

Comment 41

[Malte Jürgens [:maltejur]]

I may be mistaken here, but could it be that the changes to third_party/python/taskcluster_taskgraph have been accidentally reverted by D177802, because third_party/** should only be used read-only? As far as I can tell, the changes should be sumitted to https://github.com/taskcluster/taskgraph instead.

Comment 42

[Nicklas Boman [:smurfd]]

Hey Malte,
im not sure what that was a comment on? but yes it does look like the change to third_party/python/taskcluster_taskgraph has been reverted.
Ill try to create a PR for it on github.

you learn something new every day :)
thanks for the catch!

Comment 43

[Nicklas Boman [:smurfd]]

PR created : https://github.com/taskcluster/taskgraph/pull/291

Comment 44

[Alex Hochheiden [:ahochheiden] [he/him]]

(In reply to Malte Jürgens from comment #41)

I may be mistaken here, but could it be that the changes to third_party/python/taskcluster_taskgraph have been accidentally reverted by D177802, because third_party/** should only be used read-only? As far as I can tell, the changes should be sumitted to https://github.com/taskcluster/taskgraph instead.

Good catch! I didn't notice the third_party dir during review. The new pull request on taskcluster looks correct.

Comment 45

[Nicklas Boman [:smurfd]]

Hey @ahal

I see you are quite active on Taskgraph's github, hope its ok that i needinfo you here? Please forward if you know of someone more suited to look at https://github.com/taskcluster/taskgraph/pull/291
TL;DR i submitted a patch to modify third_party/python/taskcluster_taskgraph/ to allow firefox to build with mach from release source.
in thirdparty it gets overwritten by github when merging versions

Comment 46

[Andrew Halberstadt [:ahal]]

Thanks for the ping! I requested some changes though the approach seems fine. Feel free to reach out again if I don't notice the update.

Comment 47

[Andrew Halberstadt [:ahal]]

So it occurred to me that we might be able to avoid modifying upstream Taskgraph entirely if the code that needs these changes adds something like:

from taskgraph import parameters

def source_parameter_defaults(root_dir):
    return {
       "base_url": "SOURCE",
        ...
    }

parameters.defaults_functions = [source_parameter_defaults]

Comment 48

[Nicklas Boman [:smurfd]]

ahal and me had some chat in the pr, and decided to close it today with this follow up bug : https://bugzilla.mozilla.org/show_bug.cgi?id=1847288
were we try to avoid touching taskgraph code, and handle this in mozilla-central instead.

Please follow there if interested.