2039519 - Thunderbird build broken, port bug 2039002

Status: REOPENED

(Thunderbird :: Build Config, defect)

Description

[Kai Engert [:KaiE:]]

Thunderbird build is broken with:
ModuleNotFoundError: No module named 'comm_taskgraph'

stack:
[task 2026-05-14T09:31:56.106Z] using default project parameters; add comm-central to PER_PROJECT_PARAMETERS in /builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/decision.py to customize behavior for this project
[task 2026-05-14T09:31:56.110Z] Traceback (most recent call last):
[task 2026-05-14T09:31:56.110Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 291, in taskgraph_decision
[task 2026-05-14T09:31:56.110Z] parameters = get_decision_parameters(graph_config, options)
[task 2026-05-14T09:31:56.110Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2026-05-14T09:31:56.110Z] File "/builds/worker/checkouts/gecko/taskcluster/gecko_taskgraph/decision.py", line 436, in get_decision_parameters
[task 2026-05-14T09:31:56.110Z] find_object(graph_config["taskgraph"]["decision-parameters"])(
[task 2026-05-14T09:31:56.110Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2026-05-14T09:31:56.110Z] File "/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/taskgraph/lib/python3.11/site-packages/taskgraph/util/python_path.py", line 23, in find_object
[task 2026-05-14T09:31:56.110Z] obj = importlib.import_module(modulepath)
[task 2026-05-14T09:31:56.110Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2026-05-14T09:31:56.110Z] File "/usr/lib/python3.11/importlib/init.py", line 126, in import_module
[task 2026-05-14T09:31:56.110Z] return _bootstrap._gcd_import(name[level:], package, level)
[task 2026-05-14T09:31:56.110Z] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[task 2026-05-14T09:31:56.110Z] File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
[task 2026-05-14T09:31:56.110Z] File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
[task 2026-05-14T09:31:56.110Z] File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked
[task 2026-05-14T09:31:56.110Z] File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
[task 2026-05-14T09:31:56.110Z] File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
[task 2026-05-14T09:31:56.110Z] File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
[task 2026-05-14T09:31:56.110Z] File "<frozen importlib._bootstrap>", line 1142, in _find_and_load_unlocked

Comment 1

[Kai Engert [:KaiE:]]

Here is an AI analysis:

  What changed

  Before 0112dd536dff, mach taskgraph decision flowed like this:

  mach_commands.py → taskgraph_commands["decision"].func(options)
                   → gecko_taskgraph.main.decision(options)
                   → taskgraph_decision(options, parameters=None)
                   → TaskGraphGenerator(..., parameters=lambda gc: get_decision_parameters(gc, options))

  The TaskGraphGenerator honored graph_config["taskgraph"]["register"] — for c-c that's comm_taskgraph:register — before it evaluated the parameters lambda. comm_taskgraph.register()
  calls extend_sys_path(GECKO) (the MachEnvRequirements block at comm_taskgraph/__init__.py:24), which puts comm/taskcluster on sys.path. By the time get_decision_parameters runs and
  tries to resolve 'comm_taskgraph.decision:get_decision_parameters' via find_object, the import works.

  After 0112dd536dff, mach_commands.py short-circuits all that:

  graph_config = load_graph_config(options.get("root") or "taskcluster")
  register_gecko_taskgraph(graph_config)            # only gecko's register
  parameters = get_decision_parameters(graph_config, options)  # ← line 291

  It calls only gecko_taskgraph.register and immediately calls get_decision_parameters. The graph_config's own register: comm_taskgraph:register setting
  (comm/taskcluster/config.yml:133) is never invoked, so comm/taskcluster is not on sys.path. get_decision_parameters then hits this in gecko_taskgraph/decision.py:435-438:

  if "decision-parameters" in graph_config["taskgraph"]:
      find_object(graph_config["taskgraph"]["decision-parameters"])(  # 'comm_taskgraph.decision:get_decision_parameters'
          graph_config, parameters
      )

  → ModuleNotFoundError: No module named 'comm_taskgraph'. Exactly what the log shows.

  What to port / patch in c-c

  There's nothing to port from m-c — the commit is a c-c-hostile refactor on the m-c side. The contract it broke is: "graph_config's register hook runs before decision parameters
  resolve." C-c has two options, easiest first:

  1. Bootstrap sys.path from outside the mach process. In comm/.taskcluster.yml (around the ./mach … taskgraph decision invocation at line 255), prepend comm/taskcluster to PYTHONPATH
  so comm_taskgraph is importable before anything in mach runs. Same for taskgraph action-callback if the action path also hits this. This is a small, surgical fix.
  2. File an m-c bug to fix mach_commands.py. The right fix upstream is for the new code to also call the graph_config's own register before resolving parameters — something like:
  if "register" in graph_config["taskgraph"]:
      find_object(graph_config["taskgraph"]["register"])(graph_config)
  2. right after register_gecko_taskgraph(graph_config). That would restore parity with the old behavior for any project that uses a non-gecko register hook (i.e. c-c). If you don't
  want to wait for upstream, you can carry this as a patch in c-c's build automation, but option 1 is cleaner because it doesn't fork m-c.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/eee871a7a36c
Attempt temporary workaround. r=bustage

Comment 3

[Kai Engert [:KaiE:]]

reopening, because what was pushed is just a workaround

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/6f5dc836043b
Backed out changeset eee871a7a36c because this workaround patch didn't fix the issue. r=bustage

Comment 5

[Kai Engert [:KaiE:]]

Hello abhishekmadan, we believe that the change from bug 2039002 caused this bug.

May we ask for your help?

Comment 6

[Kai Engert [:KaiE:]]

I have a successful try run with a potential fix applied to m-c:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=2fcb7b4f9c2ed39610ecc2a653cafba202872803

I'll submit the m-c change to phab and ask for review.

Comment 7

[Kai Engert [:KaiE:]]

Patch submitted in m-c bug 2039002