Bugzilla

On Matrix :jgraham wrote:
> When I wrote that patch I was imagining we'd produce one artifact per task with a list of tests/groups to run in that task. So yeah, it doesn't quite work if we want to provide a single artifact for all tasks and then command line arguments that select specific groups from in that larger artifact

I agree that passing in the full list of test groups for the entire suite *in addition* to the list of manifests to run is pretty heavy handed. Personally I think the grouping algorithm is pretty simple, so I'd be inclined to instead:

1. Double check that they both produce identical output. Maybe move the algorithm to a place that both `moztest` and `WPT` can import from easily (I think :egao tried to import it from WPT but it wasn't available in all contexts).
2. Implement a CLI flag to run groups instead of a directories.
3. Update mozharness to pass this flag in.

On Matrix :jgraham wrote:
> When I wrote that patch I was imagining we'd produce one artifact per task with a list of tests/groups to run in that task. So yeah, it doesn't quite work if we want to provide a single artifact for all tasks and then command line arguments that select specific groups from in that larger artifact

I agree that passing in the full list of test groups for the entire suite *in addition* to the list of groups to run is pretty heavy handed. Personally I think the grouping algorithm is pretty simple, so I'd be inclined to instead:

1. Double check that they both produce identical output. Maybe move the algorithm to a place that both `moztest` and `WPT` can import from easily (I think :egao tried to import it from WPT but it wasn't available in all contexts).
2. Implement a CLI flag to run groups instead of a directories.
3. Update mozharness to pass this flag in.

On Matrix :jgraham wrote:
> When I wrote that patch I was imagining we'd produce one artifact per task with a list of tests/groups to run in that task. So yeah, it doesn't quite work if we want to provide a single artifact for all tasks and then command line arguments that select specific groups from in that larger artifact

I agree that passing in the full list of test groups for the entire suite *in addition* to the list of groups to run is pretty heavy handed. Personally I think the grouping algorithm is pretty simple and the risk of divergence is small, so I'd be inclined to instead:

1. Double check that they both produce identical output. Maybe move the algorithm to a place that both `moztest` and `WPT` can import from easily (I think :egao tried to import it from WPT but it wasn't available in all contexts).
2. Implement a CLI flag to run groups instead of a directories.
3. Update mozharness to pass this flag in.

On Matrix :jgraham wrote:
> When I wrote that patch I was imagining we'd produce one artifact per task with a list of tests/groups to run in that task. So yeah, it doesn't quite work if we want to provide a single artifact for all tasks and then command line arguments that select specific groups from in that larger artifact

I agree that passing in the full list of test groups for the entire suite *in addition* to the list of groups to run is pretty heavy handed. Personally I think the grouping algorithm is pretty simple and the risk of divergence is small, so I'd be inclined to instead:

1. Double check that they both produce identical output. Maybe move the algorithm to a place that both `moztest` and `WPT` can import from easily (I think :egao tried to import it from WPT but it wasn't available in all contexts). If that's not feasible, an integration test that makes sure they both match could also work.
2. Implement a CLI flag to run groups instead of a directories.
3. Update mozharness to pass this flag in.