1157919 - Add a --use-no-chunks option to override chunking

Status: RESOLVED WORKSFORME

(Testing :: General, defect)

Description

[Chris Manchester (limited bugmail, email directly)]

My current idea for bug 1149670 involves making selected tests always run in a single chunk to sidestep the issue of what to do about automatically selecting chunks. Re-chunking within a directory also seems risky for introducing failures due to test dependencies. It also seems that if you're selecting more tests than would fit in a chunk this isn't going to be a significant time saver over running the relevant chunks in their entirety.

Anyway, if I end up needing it I have a patch that provides an option to ignore chunking options when filtering tests.

Comment 1

[Chris Manchester (limited bugmail, email directly)]

Attachment: MozReview Request: bz://1157919/chmanchester

/r/7631 - Bug 1157919 - Add an option to override chunking and skip that step from the test filtering process.;r=ahal

Pull down this commit:

hg pull -r 51c51d0eb59064b224383c530774351e69d3f40a https://reviewboard-hg.mozilla.org/gecko/

Comment 2

[Andrew Halberstadt [:ahal]]

Comment on attachment 8597413 [details]
MozReview Request: bz://1157919/chmanchester

https://reviewboard.mozilla.org/r/7629/#review6409

::: testing/mochitest/mochitest_options.py:47
(Diff revision 1)
> +        [["--use-no-chunks"],
> +         {"action": "store_true",
> +          "dest": "use_no_chunks",
> +          "default": False,
> +          "help": "Used to override chunking options and omit chunking from test filtration",
> +          }],

Hm, I can't say I'm a fan of making config options to override other config options. I know it's more difficult, but wasn't the original strategy to have something capable of modifying the in-tree configs so we could just not do chunking in the first place? I think having the ability to remove default options will come in handy sooner or later anyway.

If that's not acceptable, I've been thinking of a way to consolidate all of the chunking options anyway. Something like --chunk-strategy={'none', 'slice', 'dir', 'runtime'} which would replace --chunk-by-dir and --chunk-by-runtime. Then --this-chunk and --total-chunks would only have an affect if --chunk-strategy was something other than 'none'.

Comment 3

[Chris Manchester (limited bugmail, email directly)]

https://reviewboard.mozilla.org/r/7629/#review6411

> Hm, I can't say I'm a fan of making config options to override other config options. I know it's more difficult, but wasn't the original strategy to have something capable of modifying the in-tree configs so we could just not do chunking in the first place? I think having the ability to remove default options will come in handy sooner or later anyway.
> 
> If that's not acceptable, I've been thinking of a way to consolidate all of the chunking options anyway. Something like --chunk-strategy={'none', 'slice', 'dir', 'runtime'} which would replace --chunk-by-dir and --chunk-by-runtime. Then --this-chunk and --total-chunks would only have an affect if --chunk-strategy was something other than 'none'.

The trouble is that chunking doesn't happen in tree. I can try to strip the arguments out in mozharness, but buildbot passes chunking stuff as separate arguments ([... "--this-chunk", "1", ...]), which is a pain in the neck. --chunk-strategy sounds ok, but --chunk-by-dir is an int that's a count of directories so we'll need to add another option to specify a number of directories to chunk by if --chunk-strategy is going to replace it.

Comment 4

[Andrew Halberstadt [:ahal]]

Right, I forgot about that. Good points, I guess all those options kind of suck..

Here's another, also not ideal option. What if instead of adding a --no-use-chunks parameter, we just passed in --this-chunk 1 --total-chunks 1? When/if chunking ever moves in-tree, we could stop that practice, but in the short term it wouldn't require any changes.

Comment 5

[Chris Manchester (limited bugmail, email directly)]

https://reviewboard.mozilla.org/r/7629/#review6413

> The trouble is that chunking doesn't happen in tree. I can try to strip the arguments out in mozharness, but buildbot passes chunking stuff as separate arguments ([... "--this-chunk", "1", ...]), which is a pain in the neck. --chunk-strategy sounds ok, but --chunk-by-dir is an int that's a count of directories so we'll need to add another option to specify a number of directories to chunk by if --chunk-strategy is going to replace it.

Wouldn't having --chunk-strategy=none resulting in --this-chunk and --total-chunks being ignored still sort be config options overriding other config options?

Comment 6

[Chris Manchester (limited bugmail, email directly)]

(In reply to Andrew Halberstadt [:ahal] from comment #4)
> Right, I forgot about that. Good points, I guess all those options kind of
> suck..
> 
> Here's another, also not ideal option. What if instead of adding a
> --no-use-chunks parameter, we just passed in --this-chunk 1 --total-chunks
> 1? When/if chunking ever moves in-tree, we could stop that practice, but in
> the short term it wouldn't require any changes.

This means rewriting the command in mozharness, which is ok, but kind a pain.

Comment 7

[Andrew Halberstadt [:ahal]]

What do you mean? As long as they get appended to the command after whatever mozharness does, those values should override the mozharness ones:

In [10]: import argparse
In [11]: p = argparse.ArgumentParser()
In [12]: p.add_argument('--foo')
In [13]: p.parse_args(['--foo', '1', '--foo', '2'])
Out[13]: Namespace(foo='2')

Or do they get prepended to the command?

Comment 8

[Chris Manchester (limited bugmail, email directly)]

Oh, here I am thinking it's an error to specify an argument multiple times. You're right, we can just append, thanks!

Comment 9

[Chris Manchester (limited bugmail, email directly)]

Don't think I'll need this then.

Comment 10

[Andrew Halberstadt [:ahal]]

Just keep in mind that appending won't override arguments with action='append'. But I don't think that will be a problem for practical purposes.

Comment 12

[Chris Manchester (limited bugmail, email directly)]

Attachment: MozReview Request: Bug 1157919 - Add an option to override chunking and skip that step from the test filtering process.;r=ahal