1590410 - 'mach try fuzzy' with show-estimates, show percentile

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Simon Fraser [:sfraser] ⌚️GMT]

Add some information to the preview window when using mach try fuzzy --show-estimates so that it tells the user how large their requested task set is compared to other users.

Comment 1

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Bug 1590410 mach try fuzzy --show-estimates, add percentile r=ahal

A requested addition to the estimates in the preview pane, showing how large the requested set of tasks is compared to everyone else's set.

Comment 2

[Andrew Halberstadt [:ahal]]

This specific change seems fine. But I think there's a larger problem with --show-estimates..

Now don't get me wrong, the feature is really cool. But I don't think it will move the needle much as long as it is opt-in. The problem is that it has some bugs that need to be ironed out before it can be enabled by default:

1. Needs to be way more performant. I have a very beefy machine, but passing in -s causes some serious lag.
2. When selecting a large number of tasks, the preview pane just goes blank and appears to hang indefinitely.
3. The summary gets hidden if selecting more tasks than fit on the screen
4. The summary is only implemented for mach try fuzzy, ideally we should show this information for all try selectors.
5. Preview pane feels very busy.

Apologies that this feedback is reaching you so late.. but my recommendation would be to focus on printing this information at push time (e.g in push.py) rather than trying to get this preview pane working smoothly. Just a simple summary that displays all this info (total compute hours, estimated completion time, etc). This should be a lot simpler, no perceptible lag, work with all selectors and be almost just as useful. Plus if we force prompts for large pushes (bug 1576671) this information will be valuable context to the user that explains why they are being asked to confirm their push. The biggest benefit of all is that this approach can be turned on by default immediately without annoying developers.

Thanks for working on this though, whether or not this ends up in the preview pane or printed during push time, it will be very valuable. I'm happy to discuss further.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4b53232e488a
mach try fuzzy --show-estimates, add percentile r=ahal

Comment 4

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Andrew Halberstadt [:ahal] from comment #2)

This specific change seems fine. But I think there's a larger problem with --show-estimates..

Now don't get me wrong, the feature is really cool. But I don't think it will move the needle much as long as it is opt-in.

I didn't want to force people into it when it was still new, but roll it out slowly and get people to try it before turning it on for everyone. That way we can get feedback while only disturbing a small number of people. The goal is definitely that it's on by default, with the ability to turn it off in case downloading the required files is not an option.

The problem is that it has some bugs that need to be ironed out before it can be enabled by default:

1. Needs to be way more performant. I have a very beefy machine, but passing in -s causes some serious lag.

Do you mean before the selector appears, or while using the selector? If beforehand, it's because we now have to generate the full/target task graph, because the task set we had before doesn't include dependencies. If it's when using the selection window, it's not something I've experienced, but I can look into it.

2. When selecting a large number of tasks, the preview pane just goes blank and appears to hang indefinitely.

There are two things at play here:

1. a 500ms timeout on the preview pane, which increasing performance will help with
2. fzf passes the preview information along on the command line, so if the command is more than getconf ARG_MAX long the preview command never gets run. This will have affected people using ctrl-a before these changes. I've got a PR in to pass the data along using stdin at https://github.com/junegunn/fzf/pull/1720

3. The summary gets hidden if selecting more tasks than fit on the screen

I could look at reserving a number of rows near the bottom, but I suspect it would break scrolling. Perhaps I should add the output at the top, if we don't move it to the push summary.

4. The summary is only implemented for mach try fuzzy, ideally we should show this information for all try selectors.

I was advised to only add it for 'mach try fuzzy' because that's the preferred submission method, and that the feature should be a carrot to get people to move to it.

5. Preview pane feels very busy.

It is, but it's a lot of information I've been asked to add, and making it clear across languages and levels of experience is not straightforward. If you have ideas for how to display the information, I'm open to suggestions.

Apologies that this feedback is reaching you so late.. but my recommendation would be to focus on printing this information at push time (e.g in push.py) rather than trying to get this preview pane working smoothly.

I suppose the trade-off there is that people don't get a chance to modify their current try submission, and have to remember to change their selection for next time. My understanding was that this was desired in the preview pane, but we could make a case for it. Perhaps if I see how much I can speed up the preview, first. Or add it to both.

Comment 5

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/4b53232e488a

Comment 6

[Andrew Halberstadt [:ahal]]

Thanks for the reply!

(In reply to Simon Fraser [:sfraser] ⌚️GMT from comment #4)

Do you mean before the selector appears, or while using the selector? If beforehand, it's because we now have to generate the full/target task graph, because the task set we had before doesn't include dependencies. If it's when using the selection window, it's not something I've experienced, but I can look into it.

It's in the selection window. Without the estimates, the preview pane updates instantaneously (even with ctrl-a). With the estimates there is a slight perceptible lag when selecting a single task and a very large lag when using ctrl-a (the more tasks being selected the larger it gets). You can reproduce by typing 'mochitest<ctrl-a> with and without estimates enabled.

I tend to care a lot about the UX of this, and from my point of view having no lag is more important than showing the estimates (especially if we can simply show the estimate after the fact). This is just my opinion and is subject to debate. That being said, ideally we can figure out why there is lag and try to get rid of it so we can have our cake and eat it too.

There are two things at play here:

1. a 500ms timeout on the preview pane, which increasing performance will help with
2. fzf passes the preview information along on the command line, so if the command is more than getconf ARG_MAX long the preview command never gets run. This will have affected people using ctrl-a before these changes. I've got a PR in to pass the data along using stdin at https://github.com/junegunn/fzf/pull/1720

Ah, glad to see you have a handle on it! Though I'd expect ensuring this doesn't happen be a blocker to enabling by default.

I could look at reserving a number of rows near the bottom, but I suspect it would break scrolling. Perhaps I should add the output at the top, if we don't move it to the push summary.

Yeah, moving it to the top is probably the easiest solution.

I was advised to only add it for 'mach try fuzzy' because that's the preferred submission method, and that the feature should be a carrot to get people to move to it.

I think whoever advised you there was slightly incorrect. We definitely don't want to support try syntax as that is deprecated and we are moving people away from it. But fuzzy is just one of many selectors that are all equally supported/recommended. Other selectors include chooser, again, coverage, release and empty (with more potentially being created in the future). In the near future we're going to try and create selectors that just do the right thing so fuzzy itself might even get de-emphasized.

It is, but it's a lot of information I've been asked to add, and making it clear across languages and levels of experience is not straightforward. If you have ideas for how to display the information, I'm open to suggestions.

Maybe we could leave the estimated completion time out of the preview and only show the total compute hours? Then we could have a richer and more properly formatted estimate at push time (see next section) that includes all the other stuff like percentiles and estimated completion.

I suppose the trade-off there is that people don't get a chance to modify their current try submission, and have to remember to change their selection for next time. My understanding was that this was desired in the preview pane, but we could make a case for it. Perhaps if I see how much I can speed up the preview, first. Or add it to both.

Just to be clear, I'm suggesting we add this info to the push summary in addition to the preview approach. That way other selectors will be able to see this info too (it's equally easy to select large number of tasks with ./mach try chooser). And the reason I was suggesting to focus on the push aspect was because I believe it will be way easier than getting the preview stuff ironed out. Once it's in the push summary, you could always come back and revisit fixing up the preview pane issues. Of course it's just a suggestion :)

Comment 7

[Simon Fraser [:sfraser] ⌚️GMT]

Preview pane improvements in https://bugzilla.mozilla.org/show_bug.cgi?id=1590820

Will make a new bug for looking at push.py