788912 - Bouncer submitter failed for Firefox 15.0 and 15.0.1

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: log

Workaround is to delete and redo the builder.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

ericz grabbed a copy of the access log for bounceradmin, which revealed that two slaves were using the API around 15:14 today. This turns out to be normal, because we have the the main bouncer_submitter and one for the euballot builds too (which is why we've only hit this on releases and not betas).

There is a regression in the euballot builder. Here's what we did for 14.0.1:

python /builds/slave/rel-m-rel-eu-bncr-sub/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 python tuxedo-add.py --config firefox-tuxedo.ini --product firefox --version 14.0.1 --tuxedo-server-url https://bounceradmin.mozilla.com/api/ --bouncer-product-name Firefox --brand-name Firefox --milestone 14.0.1 --bouncer-product-suffix EUballot --platform win32-EUballot --credentials-file BuildSlaves.py
....
Adding product: Firefox-14.0.1-EUballot
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1826" name="Firefox-14.0.1-EUballot"/></products>
Adding location for Firefox-14.0.1-EUballot, win: /firefox/releases/14.0.1/win32-EUballot/:lang/Firefox%20Setup%2014.0.1.exe

And here's what 15.0.1 did:
python /builds/slave/rel-m-rel-eu-bncr-sub/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 python tuxedo-add.py --config firefox-tuxedo.ini --product firefox --version 15.0.1 --tuxedo-server-url https://bounceradmin.mozilla.com/api/ --bouncer-product-name Firefox --brand-name Firefox --partial-version 15.0 --partial-version 14.0.1 --partial-version 13.0.1 --milestone 15.0.1 --bouncer-product-suffix EUballot --platform win32-EUballot --credentials-file BuildSlaves.py
...
Adding product: Firefox-15.0.1-EUballot
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1956" name="Firefox-15.0.1-EUballot"/></products>
Adding product: Firefox-15.0.1-Complete
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1957" name="Firefox-15.0.1-Complete"/></products>
Adding product: Firefox-15.0.1-Partial-15.0
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1958" name="Firefox-15.0.1-Partial-15.0"/></products>
Adding product: Firefox-15.0.1-Partial-14.0.1
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1960" name="Firefox-15.0.1-Partial-14.0.1"/></products>
Adding product: Firefox-15.0.1-Partial-13.0.1
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1961" name="Firefox-15.0.1-Partial-13.0.1"/></products>
Adding location for Firefox-15.0.1-EUballot, win: /firefox/releases/15.0.1/win32-EUballot/:lang/Firefox%20Setup%2015.0.1.exe
Adding location for Firefox-15.0.1-Complete, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-15.0.1.complete.mar
Adding location for Firefox-15.0.1-Partial-15.0, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-15.0-15.0.1.partial.mar
Adding location for Firefox-15.0.1-Partial-14.0.1, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-14.0.1-15.0.1.partial.mar
Adding location for Firefox-15.0.1-Partial-13.0.1, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-13.0.1-15.0.1.partial.mar

ie we're adding bogus update products for EUballot, which races the main bouncer submitter and usually wins because it has less to do (no long list of locales), then the main submitter fails.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Third time is the charm for getting the blocker bug right.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom]

This just reverts the change when multiple partial support landed, 
 http://hg.mozilla.org/build/buildbotcustom/rev/69bea3f55a3a#l3.287
using the new var to disable submitting updates for EUballot.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I've added the Firefox-15.0.1-EUballot back into bouncer, and set up a win location at
 /firefox/releases/15.0.1/win32-EUballot/:lang/Firefox%20Setup%2015.0.1.exe
so 15.0.1 should be fine now.

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 658727 [details] [diff] [review]
[buildbotcustom]

Ahhhh, now I get why it was failing. Good catch, Nick!

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 658727 [details] [diff] [review]
[buildbotcustom]

http://hg.mozilla.org/build/buildbotcustom/rev/17b47f70d9fb

Comment 7

[Armen [:armenzg]]

Live on production.