1567431 - [tracking] explore the chunkification operation

Status: NEW

(Release Engineering :: General, task)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

We want to chunkify more in order to to reduce the number of jobs, hence the deps edges in the graph. But also to be consisent with what we do in l10n repacks world. This might also help us keep fewer dummy tasks in between nodes.

This bug will include but not limited to:

• beetmover
• balrog jobs

Which is why we might keep this as tracking bug and file separate bugs for each of the above, to keep patches clean and consistent.

Comment 1

[Tom Prince [:tomprince]]

Any reason for this to be private?

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Briefly talked to Ben about this earlier today on Slack. Dropping for posterity.

bhearsum: If we chunk them the same as update verify, I think we could do it without slowing down update verify at all.
mtabara: yeah, couldn’t agree more. I believe we need more chunkification not just for slightly faster turnaround but also for more robust graphs to work with
bhearsum: Yeah. I still think we need to fix the tools to handle larger graphs/groups, but in the meantime....
mtabara @nthomas and I talked about this a few weeks back and concluded that we’d ideally chunkify top-bottom, rather than bottom up.
bhearsum: Hmm, what does that mean?
mtabara which means, ideally we chunkify the balrog-submission jobs’s deps (beetmover) as well
basically chunking all the way from repackage to leaf-tasks
bhearsum: Oh, so say you have A -> B -> C -> D. You instruct B to chunk, and you implicitly chunk C & D too?
mtabara ideally, yes. rather than start chunkification process in D
I’m not saying it’s impossible to do it the other way around, it just feels more logical and I suspect, easier from taskgraph perspective
also, balrogworkers consume manifest.json produces by dep beetmoverworkers. if we were to chunk balrog jobs, but not beetmover ones, we would have to add a layer in balrogscript to glue some of those together before submitting. As opposed to chunking top-bottom where we’d get this for free, directly in beetmover
bhearsum: I think it makes a lot of sense. I'll be curious how it looks in implementation. It's not clear to me how you know what can be similarly chunked, or what you do if you have a kind that has to collapse a bunch of tasks inbetween two kinds that can be similarly chunked.
But those are just implementation details, I like the idea a lot.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Porting over some ideas dropped in https://bugzilla.mozilla.org/show_bug.cgi?id=1567429#c12.

• Chunkification needs to be done top-bottom and we should start with repackage. This is tricky as it was initially implemented to expect a single locale and that propagated to beetmover and balrog jobs. If we manage to solve this, the downstream will probably follow more easily.

• Related to 6, Nick had an interesting idea around potentially exploring prioritizing locales, instead of simple chunking. Some locales are used more than others, so it'd maybe make sense to prioritize before the rest?

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Found in triaging. Not actively working on this, returning to the pool.