984888 - bouncer submitter should run for every build attempt of a release

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[[e-mail] (:bhearsum)]

We hit a couple of edge cases with Bouncer submission recent releases. Neither would've been an issue if the submitter had run for build2.

In Firefox 28.0, build1 had partials set to 27.0.1, 27.0, and 26.0. In build2, 25.0.1 was added as a partial. Because the bouncer submitter didn't run, we didn't get a firefox-28.0-partial-25.0.1 product+locations.

In Firefox 29.0b1 tagging failed and we had to a build2. Because the bouncer submitter didn't run at all, we didn't get any bouncer entries added.

The Buildbot scheduling changes required should be pretty trivial. Rail, I think you said that the submitter script will need to be adjusted to cope though?

Comment 1

[Rail Aliiev [:rail]]

Yeah, the script should be idempotent. We could check if the desired product already exists (and probably check the product properties, like ssl_only flag, locales).

I'd like to land the WIP refactoring patch in bug 916181 first though.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #1)
> Yeah, the script should be idempotent. We could check if the desired product
> already exists (and probably check the product properties, like ssl_only
> flag, locales).
> 
> I'd like to land the WIP refactoring patch in bug 916181 first though.

What about now? :)

Comment 4

[Rail Aliiev [:rail]]

Now!

Comment 5

[Rail Aliiev [:rail]]

Attachment: multiple_bouncer-mozharness.diff

Here goes my Friday night patch. :)

The patch should look pretty straight forward. It won't add a product which already exists. This doesn't cover the following situations:

* the list of locales changes between build1 and current build
* the list of platforms changes between build1 and current build
* some or all locations change between build1 and current build

These are very rare exceptions and instead of increasing code complexity to cover them we can just handle them manually.

The change covers the case where we have different list of previous versions (for partials) in build1 and any other build.

I I haven't tested it, but I don't mind to use it for b4. It'd be very easy to back it out and rerun.

Comment 6

[Rail Aliiev [:rail]]

Attachment: multiple_bouncer-buildbotcustom.diff

Comment 7

[Rail Aliiev [:rail]]

Example API URLs:

https://bounceradmin.mozilla.com/api/product_show?product=Firefox-35.0.1
https://bounceradmin.mozilla.com/api/product_show?product=Firefox-666

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #5)
> Created attachment 8554109 [details] [diff] [review]
> multiple_bouncer-mozharness.diff
> 
> Here goes my Friday night patch. :)
> 
> The patch should look pretty straight forward. It won't add a product which
> already exists. This doesn't cover the following situations:
> 
> * the list of locales changes between build1 and current build
> * the list of platforms changes between build1 and current build
> * some or all locations change between build1 and current build
> 
> These are very rare exceptions and instead of increasing code complexity to
> cover them we can just handle them manually.

The first two definitely seem like weird enough edge cases not worry about them. The third is probably fine too. We can always come back around and support them later if necessary.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 8554109 [details] [diff] [review]
multiple_bouncer-mozharness.diff

Review of attachment 8554109 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozharness/mozilla/bouncer/submitter.py
@@ +82,5 @@
>  
> +    def product_exists(self, product_name):
> +        self.info("Checking if %s already exists" % product_name)
> +        res = self.api_call("product_show?%s" % urllib.urlencode(product_name),
> +                            data=None)

I think this URL is wrong. You will end up with something like https://bounceradmin.mozilla.com/api/product_show?Firefox-35.0.1, which returns ALL of the products. You should be using something like https://bounceradmin.mozilla.com/api/product_show?product=Firefox-35.0.1. r=me with that fixed.

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8554109 [details] [diff] [review]
multiple_bouncer-mozharness.diff

https://hg.mozilla.org/build/mozharness/rev/048cb3cc6c61 with the fix mentioned in the comment above. s/r-/r+/ :)

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8554110 [details] [diff] [review]
multiple_bouncer-buildbotcustom.diff

https://hg.mozilla.org/build/buildbotcustom/rev/74c652e55afe

Comment 12

[Rail Aliiev [:rail]]

Attachment: mozharness-fix.diff

urlencode() is for dict, quote() is for strings

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8554866 [details] [diff] [review]
mozharness-fix.diff

remote:   https://hg.mozilla.org/build/mozharness/rev/975ff40c4360
remote:   https://hg.mozilla.org/build/mozharness/rev/6540f8c0ef26
remote:   https://hg.mozilla.org/build/mozharness/rev/6aa1325da131

Comment 14

[Rail Aliiev [:rail]]

Attachment: mozharness-fix-2.diff

take 2

Comment 15

[Jordan Lund (:jlund)]

a mozharness patch has from this bug is in production

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8554873 [details] [diff] [review]
mozharness-fix-2.diff

Seems fine to me.

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 8554873 [details] [diff] [review]
mozharness-fix-2.diff

remote:   https://hg.mozilla.org/build/mozharness/rev/ccc3f9f479d4
remote:   https://hg.mozilla.org/build/mozharness/rev/4a29b02050b9
remote:   https://hg.mozilla.org/build/mozharness/rev/a1903708e76a

Comment 18

[Rail Aliiev [:rail]]

Yay! I ran the Fennec and Firefox jobs twice and the second one did nothing:

17:00:42     INFO - Submitting to https://bounceradmin.mozilla.com/api/product_show?product=Fennec-36.0b4
17:00:42     INFO - Server response
17:00:42     INFO - <?xml version="1.0" encoding="utf-8"?><products><product id="3887" name="Fennec-36.0b4"><language locale="ach"/><language locale="af"/><language locale="ak"/><language locale="an"/><language locale="ar"/><language locale="as"/><language locale="be"/><language locale="bn-IN"/><language locale="br"/><language locale="ca"/><language locale="cs"/><language locale="cy"/><language locale="da"/><language locale="de"/><language locale="dsb"/><language locale="el"/><language locale="en-GB"/><language locale="en-ZA"/><language locale="eo"/><language locale="es-AR"/><language locale="es-ES"/><language locale="es-MX"/><language locale="et"/><language locale="eu"/><language locale="fa"/><language locale="ff"/><language locale="fi"/><language locale="fr"/><language locale="fy-NL"/><language locale="ga-IE"/><language locale="gd"/><language locale="gl"/><language locale="gu-IN"/><language locale="he"/><language locale="hi-IN"/><language locale="hsb"/><language locale="hu"/><language locale="hy-AM"/><language locale="id"/><language locale="is"/><language locale="it"/><language locale="ja"/><language locale="ja-JP-mac"/><language locale="kk"/><language locale="km"/><language locale="kn"/><language locale="ko"/><language locale="lg"/><language locale="lij"/><language locale="lt"/><language locale="lv"/><language locale="mai"/><language locale="ml"/><language locale="mr"/><language locale="ms"/><language locale="nb-NO"/><language locale="nl"/><language locale="nn-NO"/><language locale="nso"/><language locale="or"/><language locale="pa-IN"/><language locale="pl"/><language locale="pt-BR"/><language locale="pt-PT"/><language locale="ro"/><language locale="ru"/><language locale="sk"/><language locale="sl"/><language locale="son"/><language locale="sq"/><language locale="sr"/><language locale="sv-SE"/><language locale="sw"/><language locale="ta"/><language locale="te"/><language locale="th"/><language locale="tr"/><language locale="uk"/><language locale="vi"/><language locale="wo"/><language locale="xh"/><language locale="zh-CN"/><language locale="zh-TW"/><language locale="zu"/></product></products>
17:00:42     INFO - Products found: 1
17:00:42  WARNING - Product Fennec-36.0b4 already exists. Skipping...

and similar for Firefox