Update verification: FAIL: no partial update found for Firefox 53.0b2, ur locale.



Release Engineering
Release Automation: Other
a year ago
9 months ago


(Reporter: jlorenzo, Unassigned)


(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)




a year ago
Both linux[1] and linux64[2] are failing with the same error:

> Retrieving 'https://aus5.mozilla.org/update/3/Firefox/53.0/20170307144748/Linux_x86-gcc3/ur/beta-cdntest/default/default/default/update.xml?force=1' from cache...
> Got this response:
> <?xml version="1.0"?>
> <updates>
>     <update type="minor" displayVersion="53.0 Beta 2" appVersion="53.0" platformVersion="53.0" buildID="20170313224914" detailsURL="https://www.mozilla.org/ur/firefox/53.0/releasenotes/">
>         <patch type="complete" URL="http://download.mozilla.org/?product=firefox-53.0b2-complete&amp;os=linux&amp;lang=ur&amp;force=1" hashFunction="sha512" hashValue="69b1d94d0a466d55cc7ce5aff764a7d9f6e25f14861c537cd1f4a71df3bd7d111d038cd542b0b552cece4e76c45872f3447517c9abba6ca33c47488ec36d236f" size="60389447"/>
>     </update>
> </updates>
> FAIL: no partial update found for https://aus5.mozilla.org/update/3/Firefox/53.0/20170307144748/Linux_x86-gcc3/ur/beta-cdntest/default/default/default/update.xml?force=1

At a first glance, it seems Balrog didn't get the the partial for the ur locale.

[1] https://tools.taskcluster.net/task-group-inspector/#/awxtNzlXS5KNolDi-VqvKA/lfwwmce_Q7aavDsQAT5GUw?_k=awbjdp
[2] https://tools.taskcluster.net/task-group-inspector/#/awxtNzlXS5KNolDi-VqvKA/ME6FiGa-THyezQZpYxWEXQ?_k=0pggdc

Comment 1

a year ago
Interestingly, the partial submission task[1], failed to upload the partial at the first attempt, but seem to have recovered. I wonder what are the logs on Balrog's side.

> 2017-03-14 03:43:31,892 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur
> 2017-03-14 03:43:31,892 - DEBUG - Data sent: None
> 2017-03-14 03:43:31,893 - INFO - Starting new HTTP connection (1): balrog
> 2017-03-14 03:43:31,989 - DEBUG - "GET /api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur HTTP/1.1" 404 81
> 2017-03-14 03:43:31,990 - ERROR - Caught HTTPError: ["Couldn't find locale identified by: Firefox-53.0b2-build1, Linux_x86-gcc3, ur"]
> 2017-03-14 03:43:31,990 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur", "timestamp": 1489463011.990695, "method": "GET", "elapsed_secs": 0.0981588363647461, "status_code": 404}
> 2017-03-14 03:43:31,991 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-53.0b2-build1
> 2017-03-14 03:43:31,991 - DEBUG - Data sent: None
> 2017-03-14 03:43:32,114 - DEBUG - "HEAD /api/releases/Firefox-53.0b2-build1 HTTP/1.1" 200 0
> 2017-03-14 03:43:32,115 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-53.0b2-build1", "timestamp": 1489463012.115007, "method": "HEAD", "elapsed_secs": 0.12292695045471191, "status_code": 200}
> 2017-03-14 03:43:32,115 - DEBUG - Balrog request to http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur
> 2017-03-14 03:43:32,115 - DEBUG - Data sent: {'data_version': '639', 'product': u'Firefox', 'hashFunction': 'sha512', 'data': '{"buildID": "20170313224914", "appVersion": "53.0", "displayVersion": "53.0 Beta 2", "partials": [{"hashValue": "730a0ff513b6fe689fe7d2b653ae7c214614ad60782ca2d4e9974154b470ba57d2f544e341ac8c452631ef684503c562e1f0e1ff61b55a66e158d8de770d8aa6", "from": "Firefox-53.0b1-build1", "filesize": 9489481}], "platformVersion": "53.0", "completes": [{"hashValue": "69b1d94d0a466d55cc7ce5aff764a7d9f6e25f14861c537cd1f4a71df3bd7d111d038cd542b0b552cece4e76c45872f3447517c9abba6ca33c47488ec36d236f", "from": "*", "filesize": 60389447}]}', 'schema_version': '4'}
> 2017-03-14 03:43:32,476 - DEBUG - "PUT /api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur HTTP/1.1" 201 25
> 2017-03-14 03:43:32,477 - DEBUG - REQUEST STATS: {"url": "http://balrog/api/releases/Firefox-53.0b2-build1/builds/Linux_x86-gcc3/ur", "timestamp": 1489463012.47758, "method": "PUT", "elapsed_secs": 0.362213134765625, "status_code": 201}

[1] https://tools.taskcluster.net/task-group-inspector/#/awxtNzlXS5KNolDi-VqvKA/_NQx9sQbTQucDAw_0uD-Qg?_k=qx2101
"ur" is a new locale since 52.0b9, see http://archive.mozilla.org/pub/firefox/candidates/52.0b9-candidates/build2/linux-i686/ and 52.0b9 is listed in partials. Our automation doesn't generate any partials if we don't have the "from" in one of the partials for a particular locale. So, even though the behaviour is suboptimal, it's expected and works around our assumptions in automation.
Last Resolved: a year ago
Resolution: --- → INCOMPLETE

Comment 3

a year ago
Hm, should the automation compare the locale list from the previous beta and this one?
IIRC the automation creates an intersection of locales for every previous build specified in partials. And we are probably lazy enough to add another loop or something to actually iterate over every locale in every partial.
Looking back over the logs I'd say 'ur' was new in 53.0b1 and the errors started in 53.0b2. Digging into Balrog the problem is that ur is not present in the Firefox-53.0b1-build1 release. Balrog needs a buildID to serve a partial from 53.0b1 to something else, and we normally get that there by submitting the complete update. So we get failures in one update verify chunk per platform, and final verification, where the update.xml contains a complete update but not a partial for requests for 53.0b1.

The releasetask template  https://github.com/mozilla/releasetasks/blob/14c3d03edb691660dc0516a5d8c4ba5bc4eb49e6/releasetasks/templates/firefox/l10n.yml.tmpl#L155  seems to look at the intersection of locales, so there's no funsize-beetmover-publish_to_balrog triplet for ur. To fix this we'd need to publish the complete somehow. 

That said, the pain should stop for 'ur' after 53.0b5 (ie now), because in 53.0b6 the partials will be b5, b4, and b2 (we skipped b3).
We hit this in bug 1271614
See Also: → bug 1271614


a year ago
See Also: → bug 1358177


9 months ago
Blocks: 1389478
You need to log in before you can comment on or make changes to this bug.