1333394 - beetmover-build bustage Chain of Trust verification error!

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

Hit m-c a lot in https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=8ff550409e1d1f8b54f6f7f115545dbef857be0b

like https://tools.taskcluster.net/task-inspector/#MWv-OuoGQr-SCcAKuR8TCg/0

2017-01-24T12:08:23     INFO - Verifying beetmover:decision GaKA95I9Rh-VueTOv88aMA command...

2017-01-24T12:08:23     INFO - Verifying beetmover:build M9i2dRuBTtCuRl5Jde4f_A as a build task...

2017-01-24T12:08:23     INFO - Verifying beetmover:build:docker-image K6gdd6uVT2aFlPi-TlRBsQ as a docker-image task...

2017-01-24T12:08:23 CRITICAL - gecko-images is not a valid docker-image workerType!

2017-01-24T12:08:23 CRITICAL - Chain of Trust verification error!

Traceback (most recent call last):

  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 1219, in verify_chain_of_trust

    task_count = await verify_task_types(chain)

  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 1027, in verify_task_types

    await valid_task_types[task_type](chain, obj)

  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 927, in verify_docker_image_task

    raise_on_errors(errors)

  File "/builds/scriptworker/lib/python3.5/site-packages/scriptworker/cot/verify.py", line 200, in raise_on_errors

    raise CoTError("\n".join(errors))

scriptworker.exceptions.CoTError: 'gecko-images is not a valid docker-image workerType!'

Comment 1

[Carsten Book [:Tomcat]]

moving to blocker since this hit a lot of tier-1 jobs

Comment 2

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

Attachment: scriptworker PR

I'm not sure, but I'm under the impression the image renaming that happened 2+ weeks ago (bug 1320328) hit production. I'm double-checking that with :dustin on IRC.

If I'm correct, this simple fix in scriptworker should do it.

Comment 3

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

> 15:33:01 <•garndt> jlorenzo: the worker type name change landed on m-c on the 7th,
> so it's been using it since then at least there.  It's possible that a new image was
> built on m-c recently and that's why cot is just now seeing it

Per [1], no image build occurred between Jan 7th and yesterday. This is very likely caused by the fact we didn't make the changes of bug 1320328 in scriptworker. 

Clearing NI's and asking :jlund for a review.

[1] https://tools.taskcluster.net/index/artifacts/#docker.images.v2.level-3.desktop-build.pushdate.2017/docker.images.v2.level-3.desktop-build.pushdate.2017

Comment 4

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

Attachment: scriptworker PR - v1 branch

Comment 5

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

First PR (attachment 8829848 [details] [review]) landed in master at: https://github.com/mozilla-releng/scriptworker/commit/c5d6856a85f78ed5968c35faf845b017f858ebdc

Waiting for review aki's review to validate the newly created v1 branch.

Comment 6

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

Second PR landed in v1 branch at https://github.com/mozilla-releng/scriptworker/commit/59bc81df05094829ce5fae0b92fbd2eff31d4bac.

Scriptworker 1.0.0b8 released: https://github.com/mozilla-releng/scriptworker/releases/tag/1.0.0b8

Comment 7

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

Attachment: Bug 1333394 - Upgrade to scriptworker 1.0.0b8

This version includes https://github.com/mozilla-releng/scriptworker/pull/67

Review commit: https://reviewboard.mozilla.org/r/106870/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/106870/

Comment 8

[Johan Lorenzo [:jlorenzo] - PTO - Back August 24th]

:aki told me mozreview is slow and he r+'d the puppet patch. That's now landed on the default branch at: https://hg.mozilla.org/build/puppet/rev/568772053921da04ed3fa0ed1bb61c49630ff2ec

Aki updated the production branch at: https://hg.mozilla.org/build/puppet/rev/2acccb11d9bf . The scriptworker instances should be updated within 30 minutes.

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 8829907 [details]
Bug 1333394 - Upgrade to scriptworker 1.0.0b8

https://reviewboard.mozilla.org/r/106870/#review107914

Comment 10

[Aki Sasaki (not active)]

I retriggered a beetmover job, and it went green. Triggered new nightlies on m-c.

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8829848 [details] [review]
scriptworker PR

forgot to clear the review here after github. sigh.