Closed Bug 1159869 Opened 9 years ago Closed 9 years ago

Mozreview should provide sensible defaults for try pushes

Categories

(MozReview Graveyard :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1204592

People

(Reporter: dminor, Unassigned)

References

Details

Work is under way in bug 1149670 to create a mach command that would allow for developers to push a subset of tests to try based upon specifying a directory.

We've discussed having Autoland support sensible defaults for try jobs in the past to conserve resources. One way forward would be to build on this work inside Autoland. We could expose this functionality through mozreview or perhaps try defaulting to running the tests associated with the directories which have changes in them.

This functionality should be opt-in so that developers always have the choice of manually specifying which try jobs should be run.
It's a short trip from the tool in bug 1149670 to doing some amount of this automatically based on the files that actually changed, which sounds pretty excellent to me.

My work in progress for bug 1149670 depends on some build system stuff to resolve tests. If doing this on Autoland makes sense (I think it does, given the idea of entering a try syntax and pushing a button on mozreview to push commits to try) I should probably yank that out and resolve the tests myself.
jmaher suggested using the list of jobs at http://alertmanager.allizom.org/seta.html as defaults if no other information is available.
Another thing we need to think about is a way to trigger the "rest" of the tests. Although running certain directories/jobs will help find a errors early in a lot of cases, I might still want to run a "full" set of tests before landing. We need a button somewhere that triggers remaining chunks and discards extra flags constraining tests (right I'm putting them in the try syntax itself).
Probably the most generally useful place to implement comment 3 will be treeherder.
(In reply to Chris Manchester [:chmanchester] from comment #3)
> Another thing we need to think about is a way to trigger the "rest" of the
> tests. Although running certain directories/jobs will help find a errors
> early in a lot of cases, I might still want to run a "full" set of tests
> before landing. We need a button somewhere that triggers remaining chunks
> and discards extra flags constraining tests (right I'm putting them in the
> try syntax itself).

Is your concern about being able to run the remaining tests without having to duplicate tests that have already run? For MozReview/Autoland my thought this would be opt-in so that people could always trigger a second try run with alternate syntax, or run whatever tests they want the first time around.
(In reply to Dan Minor [:dminor] from comment #5)
> Is your concern about being able to run the remaining tests without having
> to duplicate tests that have already run? For MozReview/Autoland my thought
> this would be opt-in so that people could always trigger a second try run
> with alternate syntax, or run whatever tests they want the first time around.

I'm not worried about duplicating tests that have already run, a "second try run with alternate syntax" is exactly use case I'm thinking of, but it would be too bad if this meant waiting for new builds of the same revision. Treeherder makes sense to me because this would then be useful to anyone pushing to try, but let's put our heads together next week.
(In reply to Chris Manchester [:chmanchester] from comment #3)
> Another thing we need to think about is a way to trigger the "rest" of the
> tests. Although running certain directories/jobs will help find a errors
> early in a lot of cases, I might still want to run a "full" set of tests
> before landing. We need a button somewhere that triggers remaining chunks
> and discards extra flags constraining tests (right I'm putting them in the
> try syntax itself).

This specific feature is very easy to implement with mozci.
It knows how to determine the full set of builders for a repo and which jobs have run.

You can tell mozci to not trigger more jobs than 1 instance per builder (no duplication).
Mozci reuses existing builds if their artifacts are still on stage/s3 (not cleaned up).

From your comments you want one of these:
* run all remaining jobs
* run jobs associated to this new trychooser message
** the commit message of that push is ignore and we determine which builders are needed for the new try syntax
After discussion we decided that the best place for this is as part of the Mozreview UI for triggering try jobs so people can see the defaults prior to pushing to try. chmanchester's work has made it possible to to limit tests to specified subdirectories when pushing to try, one way ahead here would be to examine the revisions under review, identify which directories have changes in them, and automatically create try syntax based upon that.
Component: Autoland → MozReview
Priority: P1 → P3
Product: Tree Management → Developer Services
Target Milestone: --- → Future
Version: --- → unspecified
Summary: Autoland should provide sensible defaults for try pushes → Mozreview should provide sensible defaults for try pushes
See Also: → 1186637
I think the best approach here is to attempt to make use of the |mach try| command that chmanchester is developing directly.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → DUPLICATE
Target Milestone: Future → ---
Product: Developer Services → MozReview
You need to log in before you can comment on or make changes to this bug.