1587078 - mono-repo balrogscript deployment to production breaks gecko-3-balrog GCP workers

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Andreea Pavel [:apavel]]

TrTreeherder link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedJob=270281947&resultStatus=testfailed%2Cbusted%2Cexception&revision=035f52aed4427b22facfa883067e298f10ef9e97

Failure log: https://taskcluster-artifacts.net/Vr8TN67WQEGr3bEY5E4fzg/0/public/logs/chain_of_trust.log

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

We had a green balrog job running through the AM nightly at 11:20 UTC https://tools.taskcluster.net/groups/efiPWf9LSMq9J19mjfp3Tw/tasks/Ao4C9XcPQUe0fPAhJrJoyA/runs/0

At 11:50 UTC I've deployed for the first time, the balrogscript from its new home, the mono-repo. The push was https://github.com/mozilla-releng/scriptworker-scripts/pull/24 and I pushed that to production branch.
See:

• https://github.com/mozilla-releng/scriptworker-scripts/commits/production
• https://tools.taskcluster.net/groups/HbbX4LJOT0uBRuiGZXdbCQ

Something is wrong along the way, digging now.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Found the culprit, it's related to bug 1587068.

We adjusted for signingscript only, but grafted the changes to mono-repo. Then we switched over balrogscript, which still has its ed25519 key double-encoded, but uses https://github.com/mozilla-releng/scriptworker-scripts/blob/master/docker.d/init.sh#L105 in deployment.

The balrog jobs are actually working as expected, the worker is just killing the task afterwards with internal-error.

We can either:
a) patch up mono-repo for now as balrogworkers are the only ones switched to production
b) ask CloudOps to assist us directly on the secrets side.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [scriptworker-scripts] Bug 1587078 - temporarily add decoding for ED25519 key

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)

Found the culprit, it's related to bug 1587068.

We adjusted for signingscript only, but grafted the changes to mono-repo. Then we switched over balrogscript, which still has its ed25519 key double-encoded, but uses https://github.com/mozilla-releng/scriptworker-scripts/blob/master/docker.d/init.sh#L105 in deployment.

The balrog jobs are actually working as expected, the worker is just killing the task afterwards with internal-error.

We can either:
a) patch up mono-repo for now as balrogworkers are the only ones switched to production

Just did this for now to unblock. Hit another issue with mono-repo deployment (potentially a side-effect of https://github.com/mozilla-releng/scriptworker-scripts/pull/26). Working on debugging this and prepping a fix. If this takes too long, I'll push to production which I know for sure it works ...

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Will push in the morning. Evening nightlies will likely fail with Exception. Feel free to ignore, fix is coming in the morning and we'll rerun the jobs.

Comment 6

[Intermittent Failures Robot]

279 failures in 785 pushes (0.355 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-central: 279

Platform breakdown:

• linux64-shippable: 93
• linux32-shippable: 33
• osx-shippable: 102
• win64-asan-reporter: 2
• windows2012-32-shippable: 31
• windows2012-64-shippable: 15
• windows2012-aarch64-shippable: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1587078&startday=2019-10-08&endday=2019-10-08&tree=all

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Turns out we were pushing the wrong docker-tag in our Docker hub registry, hence Jenkins didn't pick that up to deploy furthermore in GCP. We were including the project name (e.g. balrogscript ) in the tag, which was incorrect. We pushed https://github.com/mozilla-releng/scriptworker-scripts/commit/c212afe7d381fedac6719b07ceebcc527c109e7d to fix this and re-deployed.

I've rerun all jobs from most recent nightly (AM UTC from 9th October in https://tools.taskcluster.net/groups/S3tAxSvvTr2WdORCBDF0mw) and they are turning back green.

We're done here, sorry again for 2 days bustage.

Comment 8

[Intermittent Failures Robot]

275 failures in 797 pushes (0.345 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 275

Platform breakdown:

• linux64-shippable: 86
• linux32-shippable: 85
• osx-shippable: 78
• windows2012-32-shippable: 22
• windows2012-64-shippable: 1
• windows2012-aarch64-shippable: 1
• linux64-asan-reporter: 1
• win64-asan-reporter: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1587078&startday=2019-10-09&endday=2019-10-09&tree=all

Comment 9

[Intermittent Failures Robot]

554 failures in 3801 pushes (0.146 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 554

Platform breakdown:

• linux64-shippable: 179
• linux32-shippable: 118
• osx-shippable: 180
• win64-asan-reporter: 3
• windows2012-32-shippable: 53
• windows2012-64-shippable: 16
• windows2012-aarch64-shippable: 4
• linux64-asan-reporter: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1587078&startday=2019-10-07&endday=2019-10-13&tree=all