Closed Bug 1580622 Opened 2 months ago Closed Last month

Geckodriver toolchain jobs don't run for source code changes

Categories

(Testing :: geckodriver, defect)

Version 3
defect
Not set

Tracking

(firefox71 fixed)

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: whimboo, Assigned: tomprince)

References

(Depends on 1 open bug)

Details

Attachments

(3 files, 1 obsolete file)

As far as I can remember we discussed that during the review process of bug 1534533, and I was told that this will happen. But when I check changesets on mozilla-central for geckodriver/webdriver/mozbase-rust specific code changes I cannot see any toolchain build task for geckodriver being run:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&revision=bdb64cf16b68c4a7212ba16aef425bce66d8f4ca

There are only the binaries as repacked and signed from the Nightly build job.

I know that those jobs will run once other jobs depend on it, but it shouldn't stop having a build at all for source code changes, or?

Mike and Alexander, did we miss something?

Flags: needinfo?(nalexander)
Flags: needinfo?(mh+mozilla)

I guess that's because nothing depends on them? Moving them to not being toolchain jobs, using their own kind, could fix that.

Flags: needinfo?(mh+mozilla) → needinfo?(mozilla)

(In reply to Mike Hommey [:glandium] from comment #1)

I guess that's because nothing depends on them? Moving them to not being toolchain jobs, using their own kind, could fix that.

Right, nothing consumes them. Trigger them via try or wire them into the signing tasks which are triggered as separate kind and we'll see builds with source changes. (And we'll see lots of duplicated signing, but c'est la vie, we see that now.)

Flags: needinfo?(nalexander)

Hm ok. So unless Nick will be first with depending on these jobs for browsertime, we will have to switch the signing from the Nightly builds to those toolchain builds. So bug 1577110 should fix that.

Depends on: 1577110
Flags: needinfo?(mozilla)
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/3f7d133280e3
Build geckodriver on push; r=nalexander
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
Assignee: nobody → mozilla

Thanks a lot, Tom! So I assume this will build geckodriver now whenever code changes are landed on integration branches and mozilla-central?

Flags: needinfo?(mozilla)

Yes. Based on changes to these files.

Flags: needinfo?(mozilla)

(In reply to Tom Prince [:tomprince] from comment #8)

Yes. Based on changes to these files.

So while I see it for try, no toolchain tasks have been run on autoland for the following commit:

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=geckodriver&revision=d31291309f8994fe59e4c2f22f826797422f57c5

Flags: needinfo?(mozilla)

Investigating this a bit, it looks like the logic for generating the toolchain digest doesn't handle the case where the files are not present due to a sparse checkout.

Either hash_paths needs to be update to deal with (maybe using logic like this, or the paths need to be added to the sparse-profile.

Flags: needinfo?(mozilla)

Tom, this goes far beyond my knowledge with TaskCluster. Would you mind filing a bug, or reopening this one so we can get this fully fixed? That would be amazing.

Flags: needinfo?(mozilla)

Right now there's no easy way to have toolchain tasks that aren't
dependencies of tasks run in response to source code changes short of
putting the resources in taskgraph's sparse profile.

As more things depend on the geckodriver toolchain tasks directly this
becomes less valuable and can be removed from the sparse profile.

Re-opening due the pending patch. Thanks Nick!

Status: RESOLVED → REOPENED
Flags: needinfo?(mozilla)
Resolution: FIXED → ---
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6cb7723c715d
Run geckodriver toolchain tasks in response to source code changes. r=tomprince
Status: REOPENED → RESOLVED
Closed: 2 months ago2 months ago
Resolution: --- → FIXED

As it looks like the geckodriver builds are getting produced for each and every commit now even without any geckodriver source code changes. That's actually not what should have happened.

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=geckodriver

Status: RESOLVED → REOPENED
Flags: needinfo?(nalexander)
Resolution: FIXED → ---

Seeing that we will have the signing jobs in CI soon (bug 1577110) maybe we simply backout the last patch, and wait for it to land? It should actually fix this problem, right?

This was an oversight. The "taskgraph" sparse-profile was extended
but not used; this should address the issue.

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #17)

Seeing that we will have the signing jobs in CI soon (bug 1577110) maybe we simply backout the last patch, and wait for it to land? It should actually fix this problem, right?

No, it won't fix the problem. There are two things going on

  1. nothing was asking for these jobs to run; that was fixed by my patch
  2. the digest being generated to determine whether to rebuild was incorrect. :nalexander's patch mostly fixed this, but the comment about more dependencies in the commit messages is incorrect. After (1), it is always being pulled into the target graph, just sometimes being optimized away.

After adding some logging, what appears to be happening is that mach_commands.pyc files are (sometimes) being created in the geckodriver directories, which cause the digest to change. I'm not sure what the right solution is. We could change the list of files to include to exclude the .pyc files, or we could change mach to not generate them.

I'm not following this closely, so apologies if this turns out to be useless advice, but if we're running mach commands in CI where we don't want to generate .pyc files we can just set PYTHONDONTWRITEBYTECODE=1 in the task environment (ignoring such files seems better but potentially more work?).

Attachment #9097408 - Attachment is obsolete: true

(In reply to Andrei Ciure[:andrei_ciure] from comment #15)

https://hg.mozilla.org/mozilla-central/rev/6cb7723c715d

This patch caused the geckodriver toolchain jobs to run for each and every build. As discussed with Nick by end of last week it is best to get this patch backed out. Forwarding needinfo to the sheriffs.

Flags: needinfo?(nalexander) → needinfo?(sheriffs)

As discussed on IRC we will not backout but land the follow-up patch instead.

Flags: needinfo?(sheriffs)
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/1e29bf90b3ea
Disable python bytecode generation in the decision task; r=nalexander

Backed out changeset 1e29bf90b3ea (Bug 1580622) for breaking retriggers, backfill and reruns on autoland

Push with problems: https://hg.mozilla.org/integration/autoland/rev/1e29bf90b3ea

Retriggers, backfill and reruns stoped working after this bug got landed on autoland

Relanded

Flags: needinfo?(mozilla)
Backout by dvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/7a8b337580fb
Backed out changeset 1e29bf90b3ea for breaking retriggers, backfill and reruns on autoland
Pushed by dvarga@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/035f52aed442
Disable python bytecode generation in the decision task; r=nalexander on a CLOSED TREE
Status: REOPENED → RESOLVED
Closed: 2 months agoLast month
Resolution: --- → FIXED

Thanks for the final fix. We no longer run the toolchain jobs for each and every commit.

Flags: needinfo?(mozilla)
You need to log in before you can comment on or make changes to this bug.