1762712 - Exception: In build kind task 'linux64/opt': voluptuous.error.MultipleInvalid: extra keys not allowed @ data['optimization']['skip-suite-only']

Status: RESOLVED FIXED

(Thunderbird :: Upstream Synchronization, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: mkmelin [at] iki.fi
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=373176285&repo=comm-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/XL-zhqA-TQS4viqRhoBShA/runs/0/artifacts/public/logs/live_backing.log

---

Should be from bug 1748929

Comment 1

[Rob Lemley [:rjl]]

It seems to be 234316412a0e922ba161c1c33a3a085fc9177a31.

IIRC, set_optimization_schema() only worked because of the module loading order. It looks like gecko_taskgraph.transforms.task is getting imported earlier now, which means it's got a different copy of OptimizationSchema that's missing the additional thunderbird_optimizations.

I'm landing a bustage-fix, but it's pretty bad.

Andrew, could you help out with set_optimization_schema() not working anymore?

Comment 2

[Rob Lemley [:rjl]]

Attachment: temp_fix.patch

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/20f64f53be8e
Temporary fix for broken set_optimization_schema(). rs=bustage-fix

Comment 4

[Intermittent Failures Robot]

1 failures in 3323 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform and build breakdown:

• gecko-decision: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1762712&startday=2022-03-28&endday=2022-04-03&tree=all

Comment 5

[Andrew Halberstadt [:ahal]]

Interesting, I don't really understand why bug 1748929 would change the import order of transforms/task.py or anything related to optimizations.. :/. Afaict that bug doesn't add any new modules to anything's import scope.

Sorry, I don't really understand the cause of the issue to be much help here.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/6a9de01129d7
Fix taskgraph actions. rs=bustage-fix CLOSED TREE DONTBUILD

Comment 7

[Rob Lemley [:rjl]]

Second patch is to fix action triggering. Merge-day work was completely busted.

Comment 8

[Intermittent Failures Robot]

1 failures in 2922 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• comm-central: 1

Platform and build breakdown:

• lint: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1762712&startday=2022-04-04&endday=2022-04-10&tree=all

Comment 9

[Rob Lemley [:rjl]]

Attachment: Bug 1762712 - Run "initial" taskgraph verifications after graph_config.register(). r=#taskgraph-reviewers

The "initial" verifications import gecko_taskgraph.transforms.job.registry.
Doing so brings in gecko.transforms.task as a depdendency, which imports
gecko_taskgraph.optimize.schema.OptimizationSchema.

Thunderbird's call to gecko_taskgraph.optimize.schema.set_optimization_schema()
needs to happen before gecko.transforms.task is loaded so that Thunderbird-
specific optimizations are registered. That happens when graph_config.register()
is called, so to fix the situation, run the "initial" verifications after
calling graph_config.register().

Comment 10

[Rob Lemley [:rjl]]

Attachment: Bug 1762712 - Remove hacky mangling of sys.modules in comm_taskgraph.register. r=#thunderbird-reviewers

The import order problem has been identified, so this horrible hack can be
removed. The m-c fix also triggers a check for job.run_using transform
functions that didn't work quite right for comm_taskgraph's toolchain scripts
so documentation files have been adjusted accordingly.

This patch depends on D152446 landing in mozilla-central.

Comment 11

[Rob Lemley [:rjl]]

Attachment: WIP: Bug 1762712 - Port bug 1779646: Reorganize taskgraph transforms docs. rs=docs-only

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/512ec3596694
Port bug 1779646: Reorganize taskgraph transforms docs. rs=docs-only DONTBUILD CLOSED TREE

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/149dba9e0526
Run "initial" taskgraph verifications after graph_config.register(). r=taskgraph-reviewers,ahal

Comment 14

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/149dba9e0526

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4115ac28079f
Remove hacky mangling of sys.modules in comm_taskgraph.register. r=mkmelin