1754496 - Merge utility modules between standalone and Gecko taskgraphs

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1754496 - Add comprehensive tests for 'util/taskcluster.py', r?#releng-reviewers!

There are a lot of subtle differences in this file between 'gecko_taskgraph'
and standalone. Before merging this, I wanted to have a good set of tests
first.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1754496 - Sync changes to requests session from gecko_taskgraph in util/taskcluster.py, r?#releng-reviewers!

Depends on D138354

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1754496 - Version bump to 1.1.6, r?#releng-reviewers!

Depends on D138389

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1754496 - Normalize root URL in util/taskcluster.py, r?#releng-reviewers!

This syncs a fix that landed in gecko_taskgraph, and ensures the user defined
environment variables are returned in the right format (e.g no trailing slash).

Depends on D138389

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/23ce5b45e83b
Add comprehensive tests for 'util/taskcluster.py', r=releng-reviewers,aki
https://hg.mozilla.org/ci/taskgraph/rev/d460bf1bf073
Sync changes to requests session from gecko_taskgraph in util/taskcluster.py, r=releng-reviewers,jcristau
https://hg.mozilla.org/ci/taskgraph/rev/a598b98cae4c
Normalize root URL in util/taskcluster.py, r=releng-reviewers,gbrown
https://hg.mozilla.org/ci/taskgraph/rev/a7ef8c8be7f7
Version bump to 1.1.6, r=releng-reviewers,jcristau

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: WIP: Bug 1754496 - Update taskgraph vendor to 1.1.6

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: WIP: Bug 1754496 - [taskgraph] Use session and url logic from vendored taskgraph in util/taskcluster.py

Depends on D138456

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1754496 - [taskgraph] Use identical functions from vendored taskgraph in util/taskcluster.py, r?#taskgraph-reviewers!

This replaces all of the functions in gecko_taskgraph/util/taskcluster.py with
the ones from the vendored taskgraph if they are identical.

Depends on D138457

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1754496 - [taskgraph] Use 'util/time.py' from vendored taskgraph, r?#taskgraph-reviewers!

Depends on D138458

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1754496 - [taskgraph] Use 'util/parameterization.py' from vendored taskgraph, r?#taskgraph-reviewers!

Depends on D138459

Comment 11

[Andrew Halberstadt [:ahal]]

One of the utility files that needs to be converted first has an isinstance(Task) in it. So we need to convert task.py first to satisfy this check. Luckily task.py doesn't depend on any other files, so is safe to merge.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a470e4f70f80
[taskgraph] Use identical functions from vendored taskgraph in util/taskcluster.py, r=taskgraph-reviewers,aki

Comment 13

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a8b384f70d6e
Backed out changeset a470e4f70f80 as requested by ahal

Comment 14

[Andrew Halberstadt [:ahal]]

This got backed out because it broke builds with --enable-bootstrap for Emilio:

 0:03.69 Traceback (most recent call last):
 0:03.69   File "/home/emilio/src/moz/gecko-5/configure.py", line 349, in <module>
 0:03.69     sys.exit(main(sys.argv))
 0:03.69   File "/home/emilio/src/moz/gecko-5/configure.py", line 131, in main
 0:03.69     sandbox.run(os.path.join(os.path.dirname(__file__), "moz.configure"))
 0:03.69   File "/home/emilio/src/moz/gecko-5/python/mozbuild/mozbuild/configure/__init__.py", line 560, in run
 0:03.69     func(*args)
 0:03.69   File "/home/emilio/src/moz/gecko-5/python/mozbuild/mozbuild/configure/__init__.py", line 611, in _value_for
 0:03.69     return self._value_for_depends(obj)
 0:03.69   File "/home/emilio/src/moz/gecko-5/python/mozbuild/mozbuild/util.py", line 1061, in method_call
 0:03.69     cache[args] = self.func(instance, *args)
 0:03.69   File "/home/emilio/src/moz/gecko-5/python/mozbuild/mozbuild/configure/__init__.py", line 620, in _value_for_depends
 0:03.69     value = obj.result()
 0:03.69   File "/home/emilio/src/moz/gecko-5/python/mozbuild/mozbuild/util.py", line 1061, in method_call
 0:03.69     cache[args] = self.func(instance, *args)
 0:03.69   File "/home/emilio/src/moz/gecko-5/python/mozbuild/mozbuild/configure/__init__.py", line 157, in result
 0:03.69     return self._func(*resolved_args)
 0:03.69   File "/home/emilio/src/moz/gecko-5/python/mozbuild/mozbuild/configure/__init__.py", line 1294, in wrapped
 0:03.69     return new_func(*args, **kwargs)
 0:03.69   File "/home/emilio/src/moz/gecko-5/build/moz.configure/bootstrap.configure", line 219, in bootstrap_path
 0:03.69     raise e
 0:03.69   File "/home/emilio/src/moz/gecko-5/build/moz.configure/bootstrap.configure", line 214, in bootstrap_path
 0:03.69     if not try_bootstrap(os.path.exists(path)):
 0:03.69   File "/home/emilio/src/moz/gecko-5/build/moz.configure/bootstrap.configure", line 178, in try_bootstrap
 0:03.69     task_id = IndexSearch().should_replace_task(task, {}, None, task["index"])
 0:03.70   File "/home/emilio/src/moz/gecko-5/taskcluster/gecko_taskgraph/optimize/strategies.py", line 40, in should_replace_task
 0:03.70     task_id = find_task_id(index_path)
 0:03.70   File "/home/emilio/src/moz/gecko-5/third_party/python/taskcluster_taskgraph/taskgraph/util/taskcluster.py", line 192, in find_task_id
 0:03.70     return response.json()["taskId"]
 0:03.70   File "/home/emilio/src/moz/gecko-5/third_party/python/requests/requests/models.py", line 900, in json
 0:03.70     return complexjson.loads(self.text, **kwargs)
 0:03.70   File "/usr/lib64/python3.10/json/__init__.py", line 346, in loads
 0:03.70     return _default_decoder.decode(s)
 0:03.70   File "/usr/lib64/python3.10/json/decoder.py", line 337, in decode
 0:03.70     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 0:03.70   File "/usr/lib64/python3.10/json/decoder.py", line 355, in raw_decode
 0:03.70     raise JSONDecodeError("Expecting value", s, err.value) from None
 0:03.70 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0

But I can't seem to reproduce myself (on Linux). Emilio, any chance you can help narrow down a STR? What's your platform, repo type, mozconfig? Are you able to reproduce via ./mach configure?

Comment 15

[Emilio Cobos Álvarez [:emilio]]

Sure. I'm on Linux, with a git-cinnabar setup. It repros without any mozconfig file at all, and it does repro with ./mach configure:

 0:00.73 /home/emilio/src/moz/gecko-5/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python /home/emilio/src/moz/gecko-5/configure.py
 0:00.81 Using Python 3.10.2 from /home/emilio/src/moz/gecko-5/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python
 0:00.82 checking for vcs source checkout... git
 0:00.85 checking for a shell... /usr/bin/sh
 0:00.90 checking for host system type... x86_64-pc-linux-gnu
 0:00.90 checking for target system type... x86_64-pc-linux-gnu
 0:01.19 checking whether cross compiling... no
 0:01.31 checking for Python 3... /home/emilio/src/moz/gecko-5/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python (3.10.2)
 0:01.31 checking for git... /usr/bin/git
 0:01.31 checking for Git version... 2.35.1
 0:01.32 checking for sparse checkout... no
 0:01.51 error: No such remote 'origin'
 0:02.64 checking for wget... /usr/bin/wget
 0:03.09 ERROR: Expecting value: line 1 column 1 (char 0)
 0:03.09 ERROR: If you can't fix the above, retry with --disable-bootstrap.

I found this and the exception above was what I got when re-raising the exception.

Comment 16

[Andrew Halberstadt [:ahal]]

Attachment: patch: Add print debugging

Hey Emilio, I'm still struggling to reproduce. I've:

1. Applied the patch to my git-cinnabar checkout
2. Used a mozconfig with nothing but --enable-bootstrap in it
3. Ran MOZBUILD_STATE_PATH=~/temp-mozbuild ./mach configure to ensure all toolchains get bootstrapped.

But still no luck. Would you mind applying this patch and pasting the output? Would you also mind rebasing onto the latest central to make sure there wasn't something else going on with autoland there?

Thanks in advance!

Comment 17

[Andrew Halberstadt [:ahal]]

I also wonder if some toolchain task had coincidentally expired around the same time that I landed this patch (but has now been re-built).. that feels very unlikely, but could be possible?

Comment 18

[Emilio Cobos Álvarez [:emilio]]

Still fails:

$ mkdir /tmp/mozbuild                                                                                                                                                                                                                    
$ export MOZBUILD_STATE_PATH=/tmp/mozbuild                                                                                                                                                                                               
$ ./mach configure                                                                                                                                                                                                                         
Creating local state directory: /tmp/mozbuild/srcdirs/gecko-3-becae2f5ffb0
created virtual environment CPython3.10.2.final.0-64 in 32ms
  creator CPython3Posix(dest=/tmp/mozbuild/srcdirs/gecko-3-becae2f5ffb0/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Collecting glean-sdk==44.0.0
  Using cached glean_sdk-44.0.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Collecting glean-parser==5.0.1
  Using cached glean_parser-5.0.1-py3-none-any.whl (88 kB)
Collecting cffi>=1.13.0
  Using cached cffi-1.15.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
Requirement already satisfied: Jinja2>=2.10.1 in ./third_party/python/Jinja2 (from glean-parser==5.0.1->glean-sdk==44.0.0) (2.11.3)
Requirement already satisfied: PyYAML>=5.3.1 in ./third_party/python/PyYAML/lib3 (from glean-parser==5.0.1->glean-sdk==44.0.0) (5.4.1)
Requirement already satisfied: jsonschema>=3.0.2 in ./third_party/python/jsonschema (from glean-parser==5.0.1->glean-sdk==44.0.0) (3.2.0)
Requirement already satisfied: diskcache>=4 in ./third_party/python/diskcache (from glean-parser==5.0.1->glean-sdk==44.0.0) (4.1.0)
Requirement already satisfied: yamllint>=1.18.0 in ./third_party/python/yamllint (from glean-parser==5.0.1->glean-sdk==44.0.0) (1.23.0)
Requirement already satisfied: appdirs>=1.4 in ./third_party/python/appdirs (from glean-parser==5.0.1->glean-sdk==44.0.0) (1.4.4)
Requirement already satisfied: Click>=7 in ./third_party/python/click (from glean-parser==5.0.1->glean-sdk==44.0.0) (7.1.2)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Requirement already satisfied: MarkupSafe>=0.23 in ./third_party/python/MarkupSafe/src (from Jinja2>=2.10.1->glean-parser==5.0.1->glean-sdk==44.0.0) (1.1.1)
Requirement already satisfied: setuptools in ./third_party/python/setuptools (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (51.2.0)
Requirement already satisfied: pyrsistent>=0.14.0 in ./third_party/python/pyrsistent (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (0.16.0)
Requirement already satisfied: six>=1.11.0 in ./third_party/python/six (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (1.13.0)
Requirement already satisfied: attrs>=17.4.0 in ./third_party/python/attrs (from jsonschema>=3.0.2->glean-parser==5.0.1->glean-sdk==44.0.0) (19.2.0)
Requirement already satisfied: pathspec>=0.5.3 in ./third_party/python/pathspec (from yamllint>=1.18.0->glean-parser==5.0.1->glean-sdk==44.0.0) (0.9.0)
Installing collected packages: pycparser, glean-parser, cffi, glean-sdk
Successfully installed cffi-1.15.0 glean-parser-5.0.1 glean-sdk-44.0.0 pycparser-2.21
Collecting psutil<=5.8.0,>=5.4.2
  Using cached psutil-5.8.0-cp310-cp310-linux_x86_64.whl
Installing collected packages: psutil
Successfully installed psutil-5.8.0
Collecting zstandard<=0.17.0,>=0.11.1
  Using cached zstandard-0.17.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB)
Installing collected packages: zstandard
Successfully installed zstandard-0.17.0
created virtual environment CPython3.10.2.final.0-64 in 13ms
  creator CPython3Posix(dest=/home/emilio/src/moz/gecko-3/obj-x86_64-pc-linux-gnu/_virtualenvs/build, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
 0:00.92 /home/emilio/src/moz/gecko-3/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python /home/emilio/src/moz/gecko-3/configure.py
 0:01.01 Using Python 3.10.2 from /home/emilio/src/moz/gecko-3/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python
 0:01.02 checking for vcs source checkout... git
 0:01.05 checking for a shell... /usr/bin/sh
 0:01.10 checking for host system type... x86_64-pc-linux-gnu
 0:01.10 checking for target system type... x86_64-pc-linux-gnu
 0:01.40 checking whether cross compiling... no
 0:01.52 checking for Python 3... /home/emilio/src/moz/gecko-3/obj-x86_64-pc-linux-gnu/_virtualenvs/build/bin/python (3.10.2)
 0:01.52 checking for git... /usr/bin/git
 0:01.52 checking for Git version... 2.35.1
 0:01.53 checking for sparse checkout... no
 0:01.77 error: No such remote 'origin'
 0:03.17 checking for wget... /usr/bin/wget
 0:03.18 ---
 0:03.18 task:
 0:03.18 { 'artifact': 'public/build/clang.tar.zst',
 0:03.18   'index': [ 'gecko.cache.level-3.toolchains.v3.linux64-clang-13.hash.13270902b44f29c046d4e1bd0f43f30f4e9b9241a94c1de5cafc8fe5883a31da']}
 0:03.18 gecko.cache.level-3.toolchains.v3.linux64-clang-13.hash.13270902b44f29c046d4e1bd0f43f30f4e9b9241a94c1de5cafc8fe5883a31da
 0:03.71 <!doctype html><html><head><meta charset="utf-8"/><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><title>Taskcluster</title><meta name="viewport" content="width=device-width,initial-scale=1"><link rel="icon" href="/faviconLogo.png"><link href="/assets/index.a0d6c3bf.css" rel="stylesheet"></head><body><div id="root"></div><script src="/assets/runtime.82af4b31.js"></script><script src="/assets/2.8061b972.js"></script><script src="/assets/index.0bee1848.js"></script></body></html>

Comment 19

[Emilio Cobos Álvarez [:emilio]]

One thing to note is that my remote is not mozilla-central, but mozilla-unified. Can that make a difference? I see the error: No such remote 'origin' which comes from the git output but if I rename my mozilla remote to origin it still fails.

Comment 20

[Emilio Cobos Álvarez [:emilio]]

The URI it's trying to reach is https://taskcluster.net/api/index/v1/task/gecko.cache.level-3.toolchains.v3.linux64-clang-13.hash.13270902b44f29c046d4e1bd0f43f30f4e9b9241a94c1de5cafc8fe5883a31da

Comment 21

[Andrew Halberstadt [:ahal]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #19)

One thing to note is that my remote is not mozilla-central, but mozilla-unified. Can that make a difference? I see the error: No such remote 'origin' which comes from the git output but if I rename my mozilla remote to origin it still fails.

I think this is unrelated and a red herring (I get this error too as I had renamed my remote to reproduce a prior bug). It's coming from a git subcommand that fails and then which we except: pass. We should capture stderr of that subcommand so it doesn't get displayed.

(In reply to Emilio Cobos Álvarez (:emilio) from comment #20)

The URI it's trying to reach is https://taskcluster.net/api/index/v1/task/gecko.cache.level-3.toolchains.v3.linux64-clang-13.hash.13270902b44f29c046d4e1bd0f43f30f4e9b9241a94c1de5cafc8fe5883a31da

Interesting, looks like it's trying to use the taskcluster proxy. Maybe one of us has an environment variable set that is causing the different behaviours here. I'll keep poking, the output is helpful.

Comment 22

[Andrew Halberstadt [:ahal]]

Ah yes, when I unset TASKCLUSTER_ROOT_URL I can reproduce. Thanks, should be able to narrow down on it from here.

Comment 23

[Andrew Halberstadt [:ahal]]

Ok issue found and fixed, will re-land again in a bit. I thought I had already solved this, so either it was a bad rebase or I just meant to solve it but forgot. I've removed the TASKCLSUTER_ROOT_URL env from my profile file as well so I don't miss something like this in the future.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3867035dfc22
[taskgraph] Use identical functions from vendored taskgraph in util/taskcluster.py, r=taskgraph-reviewers,aki

Comment 25

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/3867035dfc22

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bdfd3a19aa49
[taskgraph] Use 'util/time.py' from vendored taskgraph, r=taskgraph-reviewers,aki

Comment 27

[Natalia Csoregi [:nataliaCs]]

Backed out for py3 failures on test_util_parameterization.py

• backout: https://hg.mozilla.org/integration/autoland/rev/9e1ab5c7ffc2e75b44ac5c0a8e7e4793994222bf
• push: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=YEgqQc-jS72cBW_Ox_IrMw.0&revision=bdfd3a19aa49abbb139b5e19dff98269fe37a0c4
• failure log: https://treeherder.mozilla.org/logviewer?job_id=369698692&repo=autoland&lineNumber=1067

[task 2022-03-02T17:03:42.985Z] 
[task 2022-03-02T17:03:42.985Z] =============================== warnings summary ===============================
[task 2022-03-02T17:03:42.985Z] taskcluster/gecko_taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-web-platform-tests]
[task 2022-03-02T17:03:42.985Z] taskcluster/gecko_taskgraph/test/test_util_chunking.py::test_get_manifests[platform0-web-platform-tests]
[task 2022-03-02T17:03:42.985Z]   /builds/worker/checkouts/gecko/testing/web-platform/manifestupdate.py:178: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
[task 2022-03-02T17:03:42.985Z]     parser = configparser.SafeConfigParser()
[task 2022-03-02T17:03:42.985Z] 
[task 2022-03-02T17:03:42.985Z] -- Docs: https://docs.pytest.org/en/latest/warnings.html
[task 2022-03-02T17:03:42.985Z] =================== 193 passed, 2 warnings in 48.75 seconds ====================
[task 2022-03-02T17:03:42.985Z] 
[task 2022-03-02T17:03:42.985Z] 
[task 2022-03-02T17:03:42.985Z] Tests Completed: 100%|██████████| 30/30 [00:55<00:00,  8.95s/Test]
[task 2022-03-02T17:03:42.985Z]                                                                   
[task 2022-03-02T17:03:42.985Z] /builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/test/test_util_parameterization.py
[task 2022-03-02T17:03:42.985Z] ============================= test session starts ==============================
[task 2022-03-02T17:03:42.985Z] platform linux -- Python 3.6.9, pytest-4.6.6, py-1.5.4, pluggy-0.13.1 -- /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/python-test/bin/python
[task 2022-03-02T17:03:42.985Z] rootdir: /builds/worker/checkouts/gecko, inifile: /builds/worker/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2022-03-02T17:03:42.985Z] collecting ... collected 20 items
[task 2022-03-02T17:03:42.985Z] 
[task 2022-03-02T17:03:42.985Z] taskcluster/gecko_taskgraph/test/test_util_parameterization.py::TestTimestamps::test_appears_with_other_keys PASSED
[task 2022-03-02T17:03:42.985Z] taskcluster/gecko_taskgraph/test/test_util_parameterization.py::TestTimestamps::test_buried_replacement TEST-UNEXPECTED-FAIL
[task 2022-03-02T17:03:42.985Z] taskcluster/gecko_taskgraph/test/test_util_parameterization.py::TestTimestamps::test_no_change PASSED

Comment 28

[Andrew Halberstadt [:ahal]]

Oops, I accidentally fixed this in the second revision instead of the first and then landed the first without the second.

Comment 29

[Andrew Halberstadt [:ahal]]

There are other utility files yet to merge, but I'm going to close this bug out for now as I have no further plans to work on this for awhile and open new ones for the remaining utility files.

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b94d6b76157a
[taskgraph] Use 'util/time.py' from vendored taskgraph, r=taskgraph-reviewers,aki
https://hg.mozilla.org/integration/autoland/rev/ab74b43937a3
[taskgraph] Use 'util/parameterization.py' from vendored taskgraph, r=taskgraph-reviewers,aki

Comment 31

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/b94d6b76157a
https://hg.mozilla.org/mozilla-central/rev/ab74b43937a3