1347100 - Update verification: FAIL: no partial update found for Firefox 53.0b2, ur locale.

Status: RESOLVED INCOMPLETE

(Release Engineering :: Release Automation, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Both linux[1] and linux64[2] are failing with the same error:

> Retrieving 'https://aus5.mozilla.org/update/3/Firefox/53.0/20170307144748/Linux_x86-gcc3/ur/beta-cdntest/default/default/default/update.xml?force=1' from cache...
> Got this response:
> <?xml version="1.0"?>
> <updates>
>     <update type="minor" displayVersion="53.0 Beta 2" appVersion="53.0" platformVersion="53.0" buildID="20170313224914" detailsURL="https://www.mozilla.org/ur/firefox/53.0/releasenotes/">
>         <patch type="complete" URL="http://download.mozilla.org/?product=firefox-53.0b2-complete&amp;os=linux&amp;lang=ur&amp;force=1" hashFunction="sha512" hashValue="69b1d94d0a466d55cc7ce5aff764a7d9f6e25f14861c537cd1f4a71df3bd7d111d038cd542b0b552cece4e76c45872f3447517c9abba6ca33c47488ec36d236f" size="60389447"/>
>     </update>
> </updates>
> 
> FAIL: no partial update found for https://aus5.mozilla.org/update/3/Firefox/53.0/20170307144748/Linux_x86-gcc3/ur/beta-cdntest/default/default/default/update.xml?force=1

At a first glance, it seems Balrog didn't get the the partial for the ur locale.

[1] https://tools.taskcluster.net/task-group-inspector/#/awxtNzlXS5KNolDi-VqvKA/lfwwmce_Q7aavDsQAT5GUw?_k=awbjdp
[2] https://tools.taskcluster.net/task-group-inspector/#/awxtNzlXS5KNolDi-VqvKA/ME6FiGa-THyezQZpYxWEXQ?_k=0pggdc

Comment 1

[Johan Lorenzo [:jlorenzo]]

Interestingly, the partial submission task[1], failed to upload the partial at the first attempt, but seem to have recovered. I wonder what are the logs on Balrog's side.


> 2017-03-14 03:43:31,892 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur
> 2017-03-14 03:43:31,892 - DEBUG - Data sent: None
> 2017-03-14 03:43:31,893 - INFO - Starting new HTTP connection (1): balrog
> 2017-03-14 03:43:31,989 - DEBUG - "GET /api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur HTTP/1.1" 404 81
> 2017-03-14 03:43:31,990 - ERROR - Caught HTTPError: ["Couldn't find locale identified by: Firefox-53.0b2-build1, Linux_x86-gcc3, ur"]
> 2017-03-14 03:43:31,990 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur", "timestamp": 1489463011.990695, "method": "GET", "elapsed_secs": 0.0981588363647461, "status_code": 404}
> 2017-03-14 03:43:31,991 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-53.0b2-build1
> 2017-03-14 03:43:31,991 - DEBUG - Data sent: None
> 2017-03-14 03:43:32,114 - DEBUG - "HEAD /api/releases/Firefox-53.0b2-build1 HTTP/1.1" 200 0
> 2017-03-14 03:43:32,115 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-53.0b2-build1", "timestamp": 1489463012.115007, "method": "HEAD", "elapsed_secs": 0.12292695045471191, "status_code": 200}
> 2017-03-14 03:43:32,115 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur
> 2017-03-14 03:43:32,115 - DEBUG - Data sent: {'data_version': '639', 'product': u'Firefox', 'hashFunction': 'sha512', 'data': '{"buildID": "20170313224914", "appVersion": "53.0", "displayVersion": "53.0 Beta 2", "partials": [{"hashValue": "730a0ff513b6fe689fe7d2b653ae7c214614ad60782ca2d4e9974154b470ba57d2f544e341ac8c452631ef684503c562e1f0e1ff61b55a66e158d8de770d8aa6", "from": "Firefox-53.0b1-build1", "filesize": 9489481}], "platformVersion": "53.0", "completes": [{"hashValue": "69b1d94d0a466d55cc7ce5aff764a7d9f6e25f14861c537cd1f4a71df3bd7d111d038cd542b0b552cece4e76c45872f3447517c9abba6ca33c47488ec36d236f", "from": "*", "filesize": 60389447}]}', 'schema_version': '4'}
> 2017-03-14 03:43:32,476 - DEBUG - "PUT /api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur HTTP/1.1" 201 25
> 2017-03-14 03:43:32,477 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur", "timestamp": 1489463012.47758, "method": "PUT", "elapsed_secs": 0.362213134765625, "status_code": 201}

[1] https://tools.taskcluster.net/task-group-inspector/#/awxtNzlXS5KNolDi-VqvKA/_NQx9sQbTQucDAw_0uD-Qg?_k=qx2101

Comment 2

[Rail Aliiev [:rail]]

"ur" is a new locale since 52.0b9, see http://archive.mozilla.org/pub/firefox/candidates/52.0b9-candidates/build2/linux-i686/ and 52.0b9 is listed in partials. Our automation doesn't generate any partials if we don't have the "from" in one of the partials for a particular locale. So, even though the behaviour is suboptimal, it's expected and works around our assumptions in automation.

Comment 3

[Aki Sasaki (not active)]

Hm, should the automation compare the locale list from the previous beta and this one?

Comment 4

[Rail Aliiev [:rail]]

IIRC the automation creates an intersection of locales for every previous build specified in partials. And we are probably lazy enough to add another loop or something to actually iterate over every locale in every partial.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Looking back over the logs I'd say 'ur' was new in 53.0b1 and the errors started in 53.0b2. Digging into Balrog the problem is that ur is not present in the Firefox-53.0b1-build1 release. Balrog needs a buildID to serve a partial from 53.0b1 to something else, and we normally get that there by submitting the complete update. So we get failures in one update verify chunk per platform, and final verification, where the update.xml contains a complete update but not a partial for requests for 53.0b1.

The releasetask template  https://github.com/mozilla/releasetasks/blob/14c3d03edb691660dc0516a5d8c4ba5bc4eb49e6/releasetasks/templates/firefox/l10n.yml.tmpl#L155  seems to look at the intersection of locales, so there's no funsize-beetmover-publish_to_balrog triplet for ur. To fix this we'd need to publish the complete somehow. 

That said, the pain should stop for 'ur' after 53.0b5 (ie now), because in 53.0b6 the partials will be b5, b4, and b2 (we skipped b3).

Comment 6

[Rail Aliiev [:rail]]

We hit this in bug 1271614