1528161 - netherlands locale fails to submit to amo.m.o for 66.0b8

Status: RESOLVED FIXED

(Cloud Services :: Operations: AMO, task, P1)

Description

[Jordan Lund (:jlund)]

this blocks the 66.0b8 release

Comment 1

[Jordan Lund (:jlund)]

cgrebs> hmm, it's not that
21:56:36 so it got auto approved and not signed
21:56:43
— •cgrebs looks at the file in more detail
21:57:42
<•cgrebs> jlund: so, actually, the langpack got signed
21:57:58
<jlund> Jordan Lund some meta data didn't get flipped?
21:58:22 to say it's signed?
21:59:08
<•cgrebs> just to be sure, I'm looking at Version 66.0buildid20190214195736
21:59:33
<jlund> Jordan Lund correct
22:00:39 if our release automation does a GET on amo, amo sends a response saying "signed: False" and then the automation fails
22:00:47 https://irccloud.mozilla.com/file/GvQlb45J/Screen%20Shot%202019-02-14%20at%209.59.54%20PM.pngScreen Shot 2019-02-14 at 9.59.54 PM.png92.77KB • image/png
22:01:52
<•cgrebs> jlund: can you check that api call again?
22:02:03 at least in the database signed is actually True now
22:02:44 generally, auto-approval and signing only happens every ~5 minutes but I think that's already baked into your pipeline afaik
22:03:43
<jlund> Jordan Lund yeah, within the task it retries 5 times with a timeout than barfs if exhausted. the entire task itself will also retry after that.
22:03:56 I can manually rerun the task. will take a few min to get to that call
22:04:49 running: https://taskcluster-web.netlify.com/tasks/Av1lNvItTBSAkq2QECWQcw
22:10:51
<•cgrebs> cool, thanks :)
22:11:32
— •cgrebs managed to free his second hand while the little one is sleeping on the arm (and wakes up immediately otherwise…) -- looking quite funny now probably :D
22:12:39
<jlund> Jordan Lund :D
22:15:00 ⇐ TreeGhost quit ([e-mail]) Client exited
22:19:40
<jlund> Jordan Lund cgrebs: nope. still failing: https://taskcluster-artifacts.net/Av1lNvItTBSAkq2QECWQcw/12/public/logs/live_backing.log
22:20:24
<•cgrebs> hmm, only the nederlands langpack, weird
22:21:13
<jlund> Jordan Lund 'signed': False,
22:21:34
<•cgrebs> yeah, saw that
22:21:41
— •cgrebs looks
22:23:40 → philomath joined ([e-mail])
22:26:33
<•cgrebs> hmm, kibana has issues for me… meh
22:28:02 jlund: eh, wait, are you signing the langpack through another pipeline before uploading to amo?
22:29:39 because the file I downloaded clearly has a mozilla certificate in it for some reason
22:30:37 not sure if that's from us though, yet, still looking
22:30:42 ⇐ philomath quit ([e-mail]) Ping timeout: 121 seconds
22:33:25
<jlund> Jordan Lund this is what I have for docs: https://github.com/mozilla-releng/releasewarrior-2.0/blob/master/docs/addons/addons.md#process
22:34:09 I don't think we sign langpacks outside of amo. Worth noting that our automation hasn't changed recently and that every other locale worked fine
22:34:23 → CosminB and safwan joined ↔ Dagger nipped out
22:46:36
<•cgrebs> Auto Approval for Nederlands (NL) Language Pack version 66.0buildid20190214195736: Was *not* auto-approved
22:46:47 okay, getting somewhere
22:50:44
<jlund> Jordan Lund if it helps, here's one that changed this beta and did not run into issues: https://addons.mozilla.org/en-US/developers/addon/8b37ee5ede5a4205986f/versions
22:54:04 → philomath joined ([e-mail])
22:55:14
<jlund> Jordan Lund cgrebs: could we delete it and try to submit again? I see there is warnings with deleting about not being able to submit the same version
22:55:47
<•cgrebs> correct, no, you can't simply submit the same version again
22:57:03 → Far^Side joined ([e-mail])
22:59:53
<•cgrebs> I have no clue thb, maybe mat or TheOne see more clearly what's wrong with this specific upload
23:00:23
<jlund> Jordan Lund :(
23:03:54
<•cgrebs> it appears there's 8 add-ons in the auto-approval queue that don't get auto approved but also don't get processed otherwise
23:04:04 that's something mat certainly knows much more about :)
23:05:14
— •cgrebs is looking at https://amo-kibana.prod.mozaws.net/_plugin/kibana/goto/865c1c136a2c51f0c783485bb2bd0eb6

Comment 2

[Jordan Lund (:jlund)]

release automation task log of failure: https://taskcluster-web.netlify.com/tasks/Av1lNvItTBSAkq2QECWQcw/runs/16/logs/https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FAv1lNvItTBSAkq2QECWQcw%2Fruns%2F16%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log

Comment 3

[Mathieu Pillard [:mat]]

A reviewer had the review page opened for the langpack, which gives them an exclusive lock over the add-on, preventing auto approval task from approving the add-on. They closed it, so it should be approved correctly now.

Comment 4

[Mathieu Pillard [:mat]]

Latest version was approved, everything back to normal.

Comment 5

[Aki Sasaki (not active)]

(In reply to Mathieu Pillard [:mat] from comment #3)

A reviewer had the review page opened for the langpack, which gives them an exclusive lock over the add-on, preventing auto approval task from approving the add-on. They closed it, so it should be approved correctly now.

Can we change this behavior? (remove the exclusive lock; they can add their review, but it doesn't prevent others from adding their review)

Comment 6

[Mathieu Pillard [:mat]]

The auto-approval task will ignore reviewer locks for language packs - this is tracked in https://github.com/mozilla/addons-server/issues/10747 which should land in production next Thursday.