1248490 - Intermittent 'invalid language code(s)' error from bounceradmin

Status: RESOLVED FIXED

(Cloud Services :: Operations: Product Delivery, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

eg: just now

14:10:39     INFO - Adding complete-mar...
14:10:39     INFO - retry: Calling _api_call with args: ('product_add/', {'languages': ['ach', 'af', 'an', 'ar', 'as', 'ast', 'az', 'be', 'bg', 'bn-BD', 'bn-IN', 'br', 'bs', 'ca', 'cs', 'cy', 'da', 'de'
, 'dsb', 'el', 'en-GB', 'en-US', 'en-ZA', 'eo', 'es-AR', 'es-CL', 'es-ES', 'es-MX', 'et', 'eu', 'fa', 'ff', 'fi', 'fr', 'fy-NL', 'ga-IE', 'gd', 'gl', 'gn', 'gu-IN', 'he', 'hi-IN', 'hr', 'hsb', 'hu', 'hy
-AM', 'id', 'is', 'it', 'ja', 'ja-JP-mac', 'kk', 'km', 'kn', 'ko', 'lij', 'lt', 'lv', 'mai', 'mk', 'ml', 'mr', 'ms', 'nb-NO', 'nl', 'nn-NO', 'or', 'pa-IN', 'pl', 'pt-BR', 'pt-PT', 'rm', 'ro', 'ru', 'si'
, 'sk', 'sl', 'son', 'sq', 'sr', 'sv-SE', 'ta', 'te', 'th', 'tr', 'uk', 'uz', 'vi', 'xh', 'zh-CN', 'zh-TW'], 'product': 'Firefox-45.0b6-Complete'}), kwargs: {}, attempt #1
14:10:39     INFO - POST data: languages=ach&languages=af&languages=an&languages=ar&languages=as&languages=ast&languages=az&languages=be&languages=bg&languages=bn-BD&languages=bn-IN&languages=br&languag
es=bs&languages=ca&languages=cs&languages=cy&languages=da&languages=de&languages=dsb&languages=el&languages=en-GB&languages=en-US&languages=en-ZA&languages=eo&languages=es-AR&languages=es-CL&languages=es-ES&languages=es-MX&languages=et&languages=eu&languages=fa&languages=ff&languages=fi&languages=fr&languages=fy-NL&languages=ga-IE&languages=gd&languages=gl&languages=gn&languages=gu-IN&languages=he&languages=hi-IN&languages=hr&languages=hsb&languages=hu&languages=hy-AM&languages=id&languages=is&languages=it&languages=ja&languages=ja-JP-mac&languages=kk&languages=km&languages=kn&languages=ko&languages=lij&languages=lt&languages=lv&languages=mai&languages=mk&languages=ml&languages=mr&languages=ms&languages=nb-NO&languages=nl&languages=nn-NO&languages=or&languages=pa-IN&languages=pl&languages=pt-BR&languages=pt-PT&languages=rm&languages=ro&languages=ru&languages=si&languages=sk&languages=sl&languages=son&languages=sq&languages=sr&languages=sv-SE&languages=ta&languages=te&languages=th&languages=tr&languages=uk&languages=uz&languages=vi&languages=xh&languages=zh-CN&languages=zh-TW&product=Firefox-45.0b6-Complete
14:10:39     INFO - Submitting to https://bounceradmin.mozilla.com/api/product_add/
14:10:39  WARNING - Cannot access https://bounceradmin.mozilla.com/api/product_add/
14:10:39  WARNING - Returned page source:
14:10:39  WARNING - <?xml version="1.0" encoding="utf-8"?><error number="103">invalid language code(s)</error>
14:10:39     INFO - retry: attempt #1 caught exception: HTTP Error 400: BAD REQUEST

We retry but hit the same error again. We also hit errors in the last release, but were able to retry to work around, see http://archive.mozilla.org/pub/firefox/candidates/45.0b5-candidates/build1/logs/release-mozilla-beta-firefox_bouncer_submitter-bm73-build1-build9.txt.gz for the log.

We haven't changed the set of locales between 45.0b5 and 45.0b6, so are there multiple machines behind bounceradmin.mozilla.com ? It smells like one or more of them don't have the current set of valid locales. That comes from a cron that pulls product-details IIRC.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I reran the job for 45.0b6 and it succeeded, although still with a lot of retries to get there.

Comment 2

[Jeremy Orem [:oremj]]

I update product details on the initial tuxedo start up. How often does it need to be updated? After running the update, I also need to restart the tuxedo workers, so there may be a small blip when this happens.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Not that often, as locales are only added infrequently - a few times a year, if that. 

How long do the workers run for ? It would be good to know if the errors are due to missing product-details on some workers, or if there is some other issue (memory leak etc etc).

Comment 4

[Jeremy Orem [:oremj]]

Since tuxedo never really changes, they last a long time.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Do you have logging which would let you see if some subset of the workers are broken ?

Comment 7

[Jeremy Orem [:oremj]]

I have a change in stage that runs update-product-details and reloads the service once per day. I'll plan on pushing this to production on Monday.

Comment 8

[Jeremy Orem [:oremj]]

I've pushed the change to production. Chances of hitting this error should now be much lower.