Closed Bug 1384210 Opened 2 years ago Closed 2 years ago

Add support to run-task for checking out comm-central branches in addition to mozilla-central branches.

Categories

(Taskcluster Graveyard :: Docker Images, enhancement)

enhancement
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla57

People

(Reporter: tomprince, Assigned: tomprince)

References

Details

Attachments

(1 file)

I'm working on porting comm-central, and as part of that I'd like to reuse m-c's docker images; Thunderbird doesn't have any dependencies beyond what Firefox needs, so it doesn't seem to make sense to maintain separate images.

I'd like to add an additional option to `run-task` to allow checking out a comm-* tree in addition to mozilla-*.
Comment on attachment 8889975 [details]
Bug 1384210 - Add comm-central checkout support to taskcluster's run-task;

https://reviewboard.mozilla.org/r/161022/#review166342

I'm not sure if it makes sense to factor out the common logic between the `GECKO_*` and `COMM_*` checkouts. I'd be happy to do that if it makes sense.

Also, this unconditionally checkout c-c after m-c. The builds I currently have running have c-c inside of m-c and there is [talk](https://bugzilla.mozilla.org/show_bug.cgi?id=648979) of doing that generally. But I'm not yet sure if that is going to work. It would be possible to make `run-task` smart enough to figure out if one path is the parent of another, and check them out in the appropriate order. But I think that change can wait until there is a need to do that.
Attachment #8889975 - Flags: review?(dustin) → review?(gps)
Here is a build that is using the new option: https://tools.taskcluster.net/task-inspector/#FRHdpgRzQZ69v0YQcQ5pow
Attachment #8889975 - Flags: review?(dustin) → review?(gps)
Summary: Add support to run-task for checking out comm-eentral branches in addition to mozilla-central branches. → Add support to run-task for checking out comm-central branches in addition to mozilla-central branches.
Blocks: 1385055
Assignee: nobody → mozilla
Comment on attachment 8889975 [details]
Bug 1384210 - Add comm-central checkout support to taskcluster's run-task;

https://reviewboard.mozilla.org/r/161022/#review167526
Attachment #8889975 - Flags: review?(gps) → review+
Pushed by ryanvm@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/bdad4ba95b6d
Add comm-central checkout support to taskcluster's run-task; r=gps
Keywords: checkin-needed
Sorry, that backout got pushed into a bunch of build bustage, so can't be used to see if the backout fixed it.

Hopefully this range has green builds at the top: https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=a9029a587fec3ad8f2d2ea7586b4d54a7c3265e5&noautoclassify&filter-searchStr=android%20mda3&group_state=expanded&tochange=c7f314bfb6f12775e54b75e0915a56bb8e77825e
There's a 0% chance this patch introduced a crash in Firefox.
I suspect that it only did so because it triggered a rebuilding of the docker image.

Here is a build that should fail in the same way:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=07f575c3b1bd7363bdb16dc103aecf7b055cc263
Non-determinism is a killer :/
It does look like triggering a new build of an identical docker image is enough to provoke a failure.
Flags: needinfo?(mozilla)
I spoke to gps about this yesterday. 

Now that task configuration is in-tree, job definitions and configs for Thunderbird should live in the comm-central repo. The are already mechanisms for layering/vendoring/merging the two repos together, and unless there's an extreme bootstrapping problem in play, we should actively avoid adding comm-central-specific code to mozilla-central.
The non-determinism in Docker image building is described in 1289812. Essentially, since we don't pin system packages, every time we generate new Docker images there's a chance a new system package will break something. This puts whoever the unfortunate person was to trigger the image rebuild in the position of hacking around it by using a known-good base image, pinning the offending package, or fixing the underlying issue.
I definitely want to put as much as possible of the Thunderbird-specific configs for in comm-central. On the other hand, I do want to avoid reinventing things that are already handled in mozilla-central.

I've been experimenting with getting taskcluster to run builds on comm-central, and have got a decision task that is picking up task definitions from comm-central (https://hg.mozilla.org/users/mozilla_hocat.ca/comm-taskcluster/file/tip/ci) using the `--root` option to `mach taskgraph`.

However, I do need some way to get both comm-central and mozilla-central checked out to be able to run `mach taskgraph` while pointing at that task definitions from comm-central. I'd like to reuse the docker images used by mozilla-central, since Thunderbird doesn't have any dependencies beyond what is needed for firefox.

I think this and Bug 1385055 are most of what I need to be able to get comm-central working on taskcluster. I know there are a couple of other places that reference GECKO_{BASE,HEAD}_{REPOSITORY,REF,REV}.

I'm sure as there will be other issues that come up, but I do definitely want to avoid comm-central specific changes to m-c. I do think this and Bug 1385055 are minimally invasive changes to get comm-central bootstraped.
Depends on: 1385950
The bug that caused the backout has been mitigated by https://hg.mozilla.org/integration/autoland/rev/f269855c3c1c
No longer depends on: 1385950
Keywords: checkin-needed
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f7c8940e634b
Add comm-central checkout support to taskcluster's run-task; r=gps
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/f7c8940e634b
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Blocks: 1391288
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.