Closed Bug 788912 Opened 10 years ago Closed 10 years ago

Bouncer submitter failed for Firefox 15.0 and 15.0.1

Categories

(Release Engineering :: Release Automation: Other, defect, P2)

x86
All
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: nthomas)

References

Details

Attachments

(2 files)

Attached file log
Workaround is to delete and redo the builder.
ericz grabbed a copy of the access log for bounceradmin, which revealed that two slaves were using the API around 15:14 today. This turns out to be normal, because we have the the main bouncer_submitter and one for the euballot builds too (which is why we've only hit this on releases and not betas).

There is a regression in the euballot builder. Here's what we did for 14.0.1:

python /builds/slave/rel-m-rel-eu-bncr-sub/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 python tuxedo-add.py --config firefox-tuxedo.ini --product firefox --version 14.0.1 --tuxedo-server-url https://bounceradmin.mozilla.com/api/ --bouncer-product-name Firefox --brand-name Firefox --milestone 14.0.1 --bouncer-product-suffix EUballot --platform win32-EUballot --credentials-file BuildSlaves.py
....
Adding product: Firefox-14.0.1-EUballot
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1826" name="Firefox-14.0.1-EUballot"/></products>
Adding location for Firefox-14.0.1-EUballot, win: /firefox/releases/14.0.1/win32-EUballot/:lang/Firefox%20Setup%2014.0.1.exe

And here's what 15.0.1 did:
python /builds/slave/rel-m-rel-eu-bncr-sub/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 python tuxedo-add.py --config firefox-tuxedo.ini --product firefox --version 15.0.1 --tuxedo-server-url https://bounceradmin.mozilla.com/api/ --bouncer-product-name Firefox --brand-name Firefox --partial-version 15.0 --partial-version 14.0.1 --partial-version 13.0.1 --milestone 15.0.1 --bouncer-product-suffix EUballot --platform win32-EUballot --credentials-file BuildSlaves.py
...
Adding product: Firefox-15.0.1-EUballot
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1956" name="Firefox-15.0.1-EUballot"/></products>
Adding product: Firefox-15.0.1-Complete
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1957" name="Firefox-15.0.1-Complete"/></products>
Adding product: Firefox-15.0.1-Partial-15.0
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1958" name="Firefox-15.0.1-Partial-15.0"/></products>
Adding product: Firefox-15.0.1-Partial-14.0.1
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1960" name="Firefox-15.0.1-Partial-14.0.1"/></products>
Adding product: Firefox-15.0.1-Partial-13.0.1
Server response:
<?xml version="1.0" encoding="utf-8"?><products><product id="1961" name="Firefox-15.0.1-Partial-13.0.1"/></products>
Adding location for Firefox-15.0.1-EUballot, win: /firefox/releases/15.0.1/win32-EUballot/:lang/Firefox%20Setup%2015.0.1.exe
Adding location for Firefox-15.0.1-Complete, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-15.0.1.complete.mar
Adding location for Firefox-15.0.1-Partial-15.0, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-15.0-15.0.1.partial.mar
Adding location for Firefox-15.0.1-Partial-14.0.1, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-14.0.1-15.0.1.partial.mar
Adding location for Firefox-15.0.1-Partial-13.0.1, win: /firefox/releases/15.0.1/update/win32-EUballot/:lang/firefox-13.0.1-15.0.1.partial.mar

ie we're adding bogus update products for EUballot, which races the main bouncer submitter and usually wins because it has less to do (no long list of locales), then the main submitter fails.
Blocks: 575317
Blocks: 775526
No longer blocks: 575317
Third time is the charm for getting the blocker bug right.
Assignee: nobody → nthomas
Blocks: 773290
No longer blocks: 775526
Priority: -- → P2
Attached patch [buildbotcustom]Splinter Review
This just reverts the change when multiple partial support landed, 
 http://hg.mozilla.org/build/buildbotcustom/rev/69bea3f55a3a#l3.287
using the new var to disable submitting updates for EUballot.
Attachment #658727 - Flags: review?(rail)
I've added the Firefox-15.0.1-EUballot back into bouncer, and set up a win location at
 /firefox/releases/15.0.1/win32-EUballot/:lang/Firefox%20Setup%2015.0.1.exe
so 15.0.1 should be fine now.
Comment on attachment 658727 [details] [diff] [review]
[buildbotcustom]

Ahhhh, now I get why it was failing. Good catch, Nick!
Attachment #658727 - Flags: review?(rail) → review+
Blocks: 788317
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Live on production.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.