1201487 - Funsize should chunkify l10n chunks

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

Funsize takes a lot of time to generate partial for Mac with 15-17 locales coming from l10n chunks. Sometimes it takes more than 2 hours.

It would be better to chunkify the chunks.

Comment 1

[Rail Aliiev [:rail]]

Henrik,

This bug will change the index routes. Instead of using the chunk number, it'll be using "{{ chunk }}_{{ subchunk }}", e.g.:

 index.funsize.v1.mozilla-central.revision.win32.a6786bf8d71d.1.2_1.balrog
 index.funsize.v1.mozilla-central.latest.win32.1.2_1.balrog

vs old

 index.funsize.v1.mozilla-central.revision.win32.a6786bf8d71d.1.2.balrog
 index.funsize.v1.mozilla-central.latest.win32.1.2.balrog

See https://tools.taskcluster.net/task-inspector/#yvIX4aCmQTK3hEa-IvtRJA/ as an example.

This should not affect your patterns unless you rely on digits or use it somewhere.

Any objections?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Fine with me. As you said, I don't actually use the chunk values so it's just a .* right now:
https://github.com/mozilla/mozmill-ci/blob/master/lib/queues.py#L143

Btw. why are sub chunks necessary and an increase of the number of chunks is not possible? Means that a chunk does not run 15 locales but only e.g. 5 of them.

Comment 3

[Rail Aliiev [:rail]]

We can increase the number of chunks, but that would add some load on hg.m.o. We try to avoid this unless it's necessary.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I see. Thanks for the explanation.

Comment 5

[Rail Aliiev [:rail]]

Attachment: update_puppet_req.diff

Comment 6

[Rail Aliiev [:rail]]

Comment on attachment 8658682 [details] [diff] [review]
update_puppet_req.diff

remote:   https://hg.mozilla.org/build/puppet/rev/101575ab2107
remote:   https://hg.mozilla.org/build/puppet/rev/bfc2d29109db

Comment 7

[Rail Aliiev [:rail]]

forgot to include more-itertools
remote:   https://hg.mozilla.org/build/puppet/rev/509e1510f700
remote:   https://hg.mozilla.org/build/puppet/rev/c374c7e7712f

Comment 8

[Rail Aliiev [:rail]]

Today's funsize tasks look much better. No more tasks longer than 2hrs. It takes ~50m on Mac for 5 locales.