1590476 - merges to comm-beta not triggering build

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Rob Lemley [:rjl]]

The past couple of merge days when pushing to comm-beta, that push is not triggering a build.
https://treeherder.mozilla.org/#/jobs?repo=comm-beta&revision=67169a45614459f525f9e2ed1e97583f6a639ba7

If it were a DONTBUILD thing, we should see a Decision task at the very least, so whatever is preventing the build happens prior to that.

Comment 1

[Rob Lemley [:rjl]]

Tom, Could I get your thoughts on this?
I'm fairly certain that it's not the content of the push itself. Looking at the Taskcluster Index at https://tools.taskcluster.net/index/comm.v2.comm-beta.pushlog-id, there's no entry for push-id #2511, which is the one in question. Are there any logs that would help? Something in ci-admin or with the hg-pushes hook perhaps?

Comment 2

[Tom Prince [:tomprince]]

https://taskcluster-ui.herokuapp.com/hooks/hg-push/comm-beta is the hook that triggers the decision image. Looking at the tasks, this is the corresponding task, and there is an error (which comes from here, though that should probably cause the task to fail).

It looks like this is the commit that is causing the failure.

Comment 3

[Rob Lemley [:rjl]]

Thanks!

Patrick, that's this commit:
hg -R beta commit --close-branch -m "closing old head a=betamerge ba=betamerge DONTBUILD CLOSED TREE"

and that "extra" head is from here it looks like:
hg -R central tag -m "Added tag BETA_BASE_$DATE for changeset $CENTRALTOBETABASEREV a=betamerge DONTBUILD CLOSED TREE" -r $CENTRALTOBETABASEREV BETA_BASE_$DATE

If that can be avoided, it's one way to fix the problem. I don/t quite get why the second head exists though?
Splitting into two pushes to hg.m.o would work too. The first push would have this angry commit in it and the second would run a build.

Wait.. I thought hg.m.o blocked commits that created multiple heads?

Comment 4

[Patrick Cloke [:clokep]]

(In reply to Rob Lemley [:rjl] from comment #3)

Patrick, that's this commit:
hg -R beta commit --close-branch -m "closing old head a=betamerge ba=betamerge DONTBUILD CLOSED TREE"

and that "extra" head is from here it looks like:
hg -R central tag -m "Added tag BETA_BASE_$DATE for changeset $CENTRALTOBETABASEREV a=betamerge DONTBUILD CLOSED TREE" -r $CENTRALTOBETABASEREV BETA_BASE_$DATE

If that can be avoided, it's one way to fix the problem. I don/t quite get why the second head exists though?

The process by which we do betas is as follows:

1. We merge c-c to c-b
2. We release c-b, uplift patches, etc. this creates a beta branch
3. Before the next merge we close the old branch
4. Go back to step 1

So the head that's being closed here is the "old" beta branch. I don't see a way around this. I'd be curious what Firefox does during merges though -- Rob had said the scripts for merges had changed pretty dramatically.

Comment 5

[Rob Lemley [:rjl]]

The new script is testing/mozharness/scripts/merge_day/gecko_migration.py and the configs for that are testing/mozharness/configs/merge_day/*.py. See https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps

It should be possible to adapt for Thunderbird. Usually the mozharness scripts work well with the whole dual-repo thing. I haven't tried though.

I'll try to dissect the process Firefox uses a little more.

Comment 6

[Rob Lemley [:rjl]]

I think I what we need to implement, I just have to find the actual commands.

Firefox does have a commit for the FIREFOX_BETA_XX_BASE that creates two heads:

Changeset: 558715 (585fe4556335) Merge autoland to mozilla-central. a=merge
User: shindli <[e-mail]>
Date: 2019-10-14 12:49:20 +0300 (9 days)
Parent: 558708 (3bdfb7bc00a0) Merge autoland to mozilla-central. a=merge
        558714 (933011b28eda) Bug 1583088 - [MIPS64]Fix visitCompareI64{AndBranch}, handle the case that rhs operand is on the stack. r=jandem …
Child: 558716 (439341f3e28c) No bug - Tagging mozilla-central 585fe45563356a10a9d53cddfeda2bd699e46dd5 with FIREFOX_BETA_71_BASE a=release DONTBUILD CLOSED TREE
      558882 (3cb6ff0bdc9d) Bug 1586225 - Correct spelling of fission, a=testonly
Tags: FIREFOX_BETA_71_BASE

Following that, there's some fun stuff happening:

c819687300edfa3a8fa3d66fac984eefb9a10c07	ffxbld — Update configs. IGNORE BROKEN CHANGESETS CLOSED TREE NO BUG a=release ba=release
2a937fe89e9e2ad33427c00233d64d5d58baa543	ffxbld — No bug - Tagging mozilla-beta 5f063fc750e9db4a19a5c62fd287f1ac4bf567d3 with FIREFOX_BETA_70_END a=release DONTBUILD CLOSED TREE
cbd9c79f3a99c6b6583d97d26b8d189c2a939fbb	ffxbld — Preserve old tags after debugsetparents. CLOSED TREE DONTBUILD a=release
0232d275b1d6a0813cb5a92b0bf956a5b38337ca	ffxbld — Merge old head via |hg debugsetparents 439341f3e28c1a0782ac0d59cd1f44cb9741de57 5f063fc750e9db4a19a5c62fd287f1ac4bf567d3|. CLOSED TREE DONTBUILD a=release
439341f3e28c1a0782ac0d59cd1f44cb9741de57	ffxbld — No bug - Tagging mozilla-central 585fe45563356a10a9d53cddfeda2bd699e46dd5 with FIREFOX_BETA_71_BASE a=release DONTBUILD CLOSED TREE

Looks like the debugsetparents command and I would guess IGNORE BROKEN CHANGESETS has some involvement as well.

Comment 7

[Rob Lemley [:rjl]]

Attachment: ignore_broken.patch

Comment 8

[Patrick Cloke [:clokep]]

Comment on attachment 9113015 [details] [diff] [review]
ignore_broken.patch

Review of attachment 9113015 [details] [diff] [review]:
-----------------------------------------------------------------

Oops, sorry I missed this today during the merge. Let's give it a try for the next one though!

Comment 9

[Patrick Cloke [:clokep]]

Let's give it a try: https://hg.mozilla.org/users/bugzilla_standard8.plus.com/drivertools/rev/09157145b6a71e4302a15c4a16f77b5d413d8e75