1781051 - .taskcluster.yml: github-push event shouldn't set the owner based on who triggered the event

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P2)

Description

[Johan Lorenzo [:jlorenzo]]

Reported error

Today, :Aryx brought this edge case to our attention: all scriptworkers tasks in this github-push graph[1] bailed on the same CoT error[2]:

2022-07-22T22:14:02 CRITICAL - scriptworker:parent NW-JHbT2RYq68z3YxGygzQ: the runtime task doesn't match any rebuilt definition!
["[('change',\n"
 "  'metadata.owner',\n"
 "  ('github-actions[bot]@users.noreply.github.com',\n"
 "   '[e-mail]')),\n"
 " ('change',\n"
 "  ['payload', 'command', 6],\n"
 "  ('taskcluster/scripts/decision-install-sdk.sh && ln -s '\n"
 "   '/builds/worker/artifacts artifacts && ~/.local/bin/taskgraph decision '\n"
 '   "--pushlog-id=\'0\' --pushdate=\'0\' --project=\'focus-android\' "\n'
 '   \'--message="" '
 "--owner=\\'github-actions[bot]@users.noreply.github.com\\' '\n"
 '   \'--level=\\\'3\\\' --base-repository="$MOBILE_BASE_REPOSITORY" \'\n'
 '   \'--head-repository="$MOBILE_HEAD_REPOSITORY" '
 '--head-ref="$MOBILE_HEAD_REF" \'\n'
 '   \'--head-rev="$MOBILE_HEAD_REV" --head-tag="$MOBILE_HEAD_TAG" \'\n'
 '   \'--repository-type="$MOBILE_REPOSITORY_TYPE" '
 "--tasks-for=\\'github-push\\' \\n',\n"
 "   'taskcluster/scripts/decision-install-sdk.sh && ln -s '\n"
 "   '/builds/worker/artifacts artifacts && ~/.local/bin/taskgraph decision '\n"
 '   "--pushlog-id=\'0\' --pushdate=\'0\' --project=\'focus-android\' "\n'
 '   \'--message="" '
 "--owner=\\'[e-mail]\\' '\n"
 '   \'--level=\\\'3\\\' --base-repository="$MOBILE_BASE_REPOSITORY" \'\n'
 '   \'--head-repository="$MOBILE_HEAD_REPOSITORY" '
 '--head-ref="$MOBILE_HEAD_REF" \'\n'
 '   \'--head-rev="$MOBILE_HEAD_REV" --head-tag="$MOBILE_HEAD_TAG" \'\n'
 '   \'--repository-type="$MOBILE_REPOSITORY_TYPE" \'\n'
 '   "--tasks-for=\'github-push\' \\n"))]']
2022-07-22T22:14:02 CRITICAL - Chain of Trust verification error!

I'm saving you from parsing the diff manually, the only bit of difference is the owner of the task (which is also passed down to the taskgraph command).

Potential explanation

This github-push event happened when :diannaS merged this PR[3] via either the Github Web UI or the Github Desktop app. I don't know if Github had a blip and got mixed up because the PR

1. was created by Github Action
2. but authored by another user
3. and merged by a 3rd user.

Probable fix

In any case, this commit is said to be committed by web-flow[5] which triggers a special if statement in CoT[6]. This special-casing was first added in [7]. Although, this time, I don't believe the fix is on the CoT side. It's likely in .taskcluster.yml and more precisely in this other if statement[8]. Instead of using ${event.pusher.email}, we may want to use ${event.head_commit.committer.email} in the regular case and ${event.head_commit.author.email} if we're dealing with web-flow again.

Implemented workaround

That said, this auto-approval change[9] will remove the need for such merges. So, I'm not sure this is the most important fix to do. I'm filing this ticket for the sake of reminding our future selves that it's known issue. Speaking of which, bug 1725665 is very similar but not identical.

[1] https://treeherder.mozilla.org/jobs?repo=focus-android&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=9f3378754095852f648002f1d3704d23b07ac671
[2] e.g.: https://firefox-ci-tc.services.mozilla.com/tasks/B2qA4ajdQVuLZK4u1AYtqA/runs/0/logs/public/logs/chain_of_trust.log#L111
[3] https://github.com/mozilla-mobile/focus-android/pull/7408
[4] https://api.github.com/repos/mozilla-mobile/focus-android/commits/9f3378754095852f648002f1d3704d23b07ac671
[5] https://api.github.com/repos/mozilla-mobile/focus-android/commits/9f3378754095852f648002f1d3704d23b07ac671
[6] https://github.com/mozilla-releng/scriptworker/blob/a79eb8d6da984ef3254be99349132f55d5d676fb/src/scriptworker/cot/verify.py#L1198-L1200
[7] https://github.com/mozilla-releng/scriptworker/issues/334
[8] https://github.com/mozilla-mobile/focus-android/blob/f65a16456d965d12eff5364193b07c3dfcd06823/.taskcluster.yml#L17-L18
[9] https://github.com/mozilla-mobile/focus-android/pull/7409

Comment 2

[Pascal Chevrel (relman team) -> :pascalc]

I hit the same issue today on the focus-android repo.
What I did was I merged the pull request (which had green CI) to bump the AC version https://github.com/mozilla-mobile/focus-android/pull/7602

Comment 3

[Julien Cristau [:jcristau]]

It looks like we get this error for all strings updates e.g. on https://github.com/mozilla-mobile/fenix/commits/releases_v105.0.0

2022-09-08T04:54:14 CRITICAL - scriptworker:parent K04ehq5TTVuW3ae58YPl-Q: the runtime task doesn't match any rebuilt definition!
["[('change',\n"
 "  'metadata.owner',\n"
 "  ('github-actions[bot]@users.noreply.github.com', '')),\n"
 " ('change',\n"
 "  ['payload', 'command', 6],\n"
 "  ('taskcluster/scripts/decision-install-sdk.sh && ln -s '\n"
 "   '/builds/worker/artifacts artifacts && ~/.local/bin/taskgraph decision '\n"
 "   '--pushlog-id=\\'0\\' --pushdate=\\'0\\' --project=\\'fenix\\' "
 '--message="" \'\n'
 '   "--owner=\'github-actions[bot]@users.noreply.github.com\' --level=\'3\' '
 '"\n'
 '   \'--base-repository="$MOBILE_BASE_REPOSITORY" \'\n'
 '   \'--head-repository="$MOBILE_HEAD_REPOSITORY" '
 '--head-ref="$MOBILE_HEAD_REF" \'\n'
 '   \'--head-rev="$MOBILE_HEAD_REV" '
 '--repository-type="$MOBILE_REPOSITORY_TYPE" \'\n'
 '   "--tasks-for=\'github-push\' \\n",\n'
 "   'taskcluster/scripts/decision-install-sdk.sh && ln -s '\n"
 "   '/builds/worker/artifacts artifacts && ~/.local/bin/taskgraph decision '\n"
 "   '--pushlog-id=\\'0\\' --pushdate=\\'0\\' --project=\\'fenix\\' "
 '--message="" \'\n'
 "   '--owner=\\'\\' --level=\\'3\\' "
 '--base-repository="$MOBILE_BASE_REPOSITORY" \'\n'
 '   \'--head-repository="$MOBILE_HEAD_REPOSITORY" '
 '--head-ref="$MOBILE_HEAD_REF" \'\n'
 '   \'--head-rev="$MOBILE_HEAD_REV" '
 '--repository-type="$MOBILE_REPOSITORY_TYPE" \'\n'
 '   "--tasks-for=\'github-push\' \\n"))]']
2022-09-08T04:54:14 CRITICAL - Chain of Trust verification error!

Comment 4

[Julien Cristau [:jcristau]]

Attachment: Fenix PR

fenix (and focus-android)'s .taskcluster.yml set the ownerEmail variable based on the event type, the event sender, (for pull requests) the PR user, and (for pushes) the pusher.
When verifying the task, chain-of-trust rebuilds the task definition using some information from the github api and some information from the original decision task's definition itself. It gets the event type, and either PR user or pusher from the decision task's definition (extra.tasks_for and metadata.owner), but tries to guess the event sender from the github API, even though the API doesn't provide that information, so it uses the committer's github login instead, with a fallback to the commit author's login.
The case where things appear to break is when 1) the committer is not set, 2) the commit author is github-actions[bot]. In that case when CoT rebuilds the decision task we end up with ownerEmail set to github-actions[bot]@users.noreply.github.com, but the original event's sender was not github-actions[bot], so the actual decision task has a different ownerEmail, and we bail on that mismatch.

Comment 5

[Julien Cristau [:jcristau]]

Attachment: Focus-android PR

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/71ff964e2251
[focus] Bug 1781051 - simplify setting ownerEmail in .taskcluster.yml
https://hg.mozilla.org/integration/autoland/rev/820150ba5443
[fenix] Bug 1781051 - simplify setting ownerEmail in .taskcluster.yml