1580622 - Geckodriver toolchain jobs don't run for source code changes

Status: RESOLVED FIXED

(Testing :: geckodriver, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As far as I can remember we discussed that during the review process of bug 1534533, and I was told that this will happen. But when I check changesets on mozilla-central for geckodriver/webdriver/mozbase-rust specific code changes I cannot see any toolchain build task for geckodriver being run:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&revision=bdb64cf16b68c4a7212ba16aef425bce66d8f4ca

There are only the binaries as repacked and signed from the Nightly build job.

I know that those jobs will run once other jobs depend on it, but it shouldn't stop having a build at all for source code changes, or?

Mike and Alexander, did we miss something?

Comment 1

[Mike Hommey [:glandium]]

I guess that's because nothing depends on them? Moving them to not being toolchain jobs, using their own kind, could fix that.

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Mike Hommey [:glandium] from comment #1)

I guess that's because nothing depends on them? Moving them to not being toolchain jobs, using their own kind, could fix that.

Right, nothing consumes them. Trigger them via try or wire them into the signing tasks which are triggered as separate kind and we'll see builds with source changes. (And we'll see lots of duplicated signing, but c'est la vie, we see that now.)

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm ok. So unless Nick will be first with depending on these jobs for browsertime, we will have to switch the signing from the Nightly builds to those toolchain builds. So bug 1577110 should fix that.

Comment 4

[Tom Prince [:tomprince]]

Attachment: Bug 1580622: Build geckodriver on push; r?glandium

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3f7d133280e3
Build geckodriver on push; r=nalexander

Comment 6

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/3f7d133280e3

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks a lot, Tom! So I assume this will build geckodriver now whenever code changes are landed on integration branches and mozilla-central?

Comment 8

[Tom Prince [:tomprince]]

Yes. Based on changes to these files.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Tom Prince [:tomprince] from comment #8)

Yes. Based on changes to these files.

So while I see it for try, no toolchain tasks have been run on autoland for the following commit:

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=geckodriver&revision=d31291309f8994fe59e4c2f22f826797422f57c5

Comment 10

[Tom Prince [:tomprince]]

Investigating this a bit, it looks like the logic for generating the toolchain digest doesn't handle the case where the files are not present due to a sparse checkout.

Either hash_paths needs to be update to deal with (maybe using logic like this, or the paths need to be added to the sparse-profile.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Tom, this goes far beyond my knowledge with TaskCluster. Would you mind filing a bug, or reopening this one so we can get this fully fixed? That would be amazing.

Comment 12

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1580622 - Run geckodriver toolchain tasks in response to source code changes. r?#build

Right now there's no easy way to have toolchain tasks that aren't
dependencies of tasks run in response to source code changes short of
putting the resources in taskgraph's sparse profile.

As more things depend on the geckodriver toolchain tasks directly this
becomes less valuable and can be removed from the sparse profile.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Re-opening due the pending patch. Thanks Nick!

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6cb7723c715d
Run geckodriver toolchain tasks in response to source code changes. r=tomprince

Comment 15

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/6cb7723c715d

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As it looks like the geckodriver builds are getting produced for each and every commit now even without any geckodriver source code changes. That's actually not what should have happened.

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=geckodriver

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Seeing that we will have the signing jobs in CI soon (bug 1577110) maybe we simply backout the last patch, and wait for it to land? It should actually fix this problem, right?

Comment 18

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1580622 - Follow-up: Use default sparse-profile. r?#build

This was an oversight. The "taskgraph" sparse-profile was extended
but not used; this should address the issue.

Comment 19

[Tom Prince [:tomprince]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #17)

Seeing that we will have the signing jobs in CI soon (bug 1577110) maybe we simply backout the last patch, and wait for it to land? It should actually fix this problem, right?

No, it won't fix the problem. There are two things going on

1. nothing was asking for these jobs to run; that was fixed by my patch
2. the digest being generated to determine whether to rebuild was incorrect. :nalexander's patch mostly fixed this, but the comment about more dependencies in the commit messages is incorrect. After (1), it is always being pulled into the target graph, just sometimes being optimized away.

After adding some logging, what appears to be happening is that mach_commands.pyc files are (sometimes) being created in the geckodriver directories, which cause the digest to change. I'm not sure what the right solution is. We could change the list of files to include to exclude the .pyc files, or we could change mach to not generate them.

Comment 20

[James Graham [:jgraham]]

I'm not following this closely, so apologies if this turns out to be useless advice, but if we're running mach commands in CI where we don't want to generate .pyc files we can just set PYTHONDONTWRITEBYTECODE=1 in the task environment (ignoring such files seems better but potentially more work?).

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrei Ciure[:andrei_ciure] from comment #15)

https://hg.mozilla.org/mozilla-central/rev/6cb7723c715d

This patch caused the geckodriver toolchain jobs to run for each and every build. As discussed with Nick by end of last week it is best to get this patch backed out. Forwarding needinfo to the sheriffs.

Comment 22

[Tom Prince [:tomprince]]

Attachment: Bug 1580622: Disable python bytecode generation in the decision task;

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As discussed on IRC we will not backout but land the follow-up patch instead.

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1e29bf90b3ea
Disable python bytecode generation in the decision task; r=nalexander

Comment 25

[Daniel Varga [:dvarga]]

Backed out changeset 1e29bf90b3ea (Bug 1580622) for breaking retriggers, backfill and reruns on autoland

Push with problems: https://hg.mozilla.org/integration/autoland/rev/1e29bf90b3ea

Retriggers, backfill and reruns stoped working after this bug got landed on autoland

Relanded

Comment 26

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7a8b337580fb
Backed out changeset 1e29bf90b3ea for breaking retriggers, backfill and reruns on autoland

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/035f52aed442
Disable python bytecode generation in the decision task; r=nalexander on a CLOSED TREE

Comment 28

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/035f52aed442

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks for the final fix. We no longer run the toolchain jobs for each and every commit.