749277 - enable multiple iterations of peptest runs on m-c, m-i, and try

Status: RESOLVED FIXED

(Testing Graveyard :: Peptest, defect)

Description

[Mark Côté [:mcote]]

Now that peptest supports --iterations to run each test multiple times (so we can average results and hopefully get more consistent numbers), we should find a good number (maybe 10) and enable this on mozilla-central, mozilla-inbout, and try, all of which are currently running each peptest exactly once per push.

Comment 1

[Mark Côté [:mcote]]

Attachment: Do 10 iterations on production systems

Okay I believe this should do it. I added an option to the config file to specify the number of iterations. I set it to 10 in prod_config.py and windows_config.py but left it out of test_config.py. I changed peptest.py to only add the option if it is set. Does this make sense? I didn't bother adding it as a command-line option to peptest.py.

Also I updated the URLs in test_config.py to the first build that supports the --iterations option.

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 622743 [details] [diff] [review]
Do 10 iterations on production systems

Perfect.

>+        if self.config.get('iterations'):
>+            cmd.extend(self._build_arg('--iterations', self.config.get('iterations')))

Since you already know that 'iterations' is in self.config, this could be

        if self.config.get('iterations'):
            cmd.extend(['--iterations', self.config['iterations']])

However, there is nothing incorrect about the way you have it.
r=me.

Comment 3

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #2)
> Comment on attachment 622743 [details] [diff] [review]
> Do 10 iterations on production systems
> 
> Perfect.
> 
> >+        if self.config.get('iterations'):
> >+            cmd.extend(self._build_arg('--iterations', self.config.get('iterations')))
> 
> Since you already know that 'iterations' is in self.config, this could be
> 
>         if self.config.get('iterations'):
>             cmd.extend(['--iterations', self.config['iterations']])
> 
> However, there is nothing incorrect about the way you have it.
> r=me.

             cmd.extend(['--iterations', str(self.config['iterations'])])

even.
So your code is more accurate than my first attempt at it :)

Comment 4

[Mark Côté [:mcote]]

(In reply to Aki Sasaki [:aki] from comment #2)
> Comment on attachment 622743 [details] [diff] [review]
> Do 10 iterations on production systems
> 
> >+        if self.config.get('iterations'):
> >+            cmd.extend(self._build_arg('--iterations', self.config.get('iterations')))
> 
> Since you already know that 'iterations' is in self.config, this could be
> 
>         if self.config.get('iterations'):
>             cmd.extend(['--iterations', self.config['iterations']])
> 
> However, there is nothing incorrect about the way you have it.
> r=me.

Actually I wanted to make iterations optional, since in peptest it defaults to 1. I didn't put it in test_config.py.

Pushed as http://hg.mozilla.org/build/mozharness/rev/5aa91ea06829. Will this be deployed automatically?

Comment 5

[Aki Sasaki (not active)]

(In reply to Mark Côté ( :mcote ) from comment #4)
> Actually I wanted to make iterations optional, since in peptest it defaults
> to 1. I didn't put it in test_config.py.

I think the |if self.config.get('iterations'):| covers making 'iterations' optional.

> Pushed as http://hg.mozilla.org/build/mozharness/rev/5aa91ea06829. Will this
> be deployed automatically?

We may change this process in the future, but currently yes.
As soon as it's pushed, subsequent peptest runs will pick that change up.

Comment 6

[Mark Côté [:mcote]]

Ah right sorry, misunderstood your comment. Anyway, thanks!