1474208 - Use median process value for AWSY base tests

Status: RESOLVED FIXED

(Testing :: AWSY, defect)

Description

[Kris Maglione [:kmag]]

There's always one content process in every base session that uses a lot more memory than the others. That leads to extreme bi-modal numbers that break our regression alerts for anything under a few hundred K.

Since we only really care about the overhead of the average base process, the extra overhead of that first process doesn't really matter. We eat it once per session, not once per process. It's a good case for the regular AWSY suite.

Taking the median value of all content processes makes that problem go away, and also increases the odds of us getting stable numbers that we can more easily spot regressions in.

Comment 1

[Kris Maglione [:kmag]]

Attachment: Bug 1474208: Use median of all process values for AWSY base content.

Review commit: https://reviewboard.mozilla.org/r/255670/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/255670/

Comment 2

[Eric Rahm [:erahm]]

Comment on attachment 8990611 [details]
Bug 1474208: Use median of all process values for AWSY base content.

https://reviewboard.mozilla.org/r/255670/#review262646

Looks good, I'd just like to cleanup the old `count` logic. We talked about possibly adding a measurement for the max as well, at most that should be a follow up if we want to do it at all.

::: testing/awsy/awsy/process_perf_data.py:41
(Diff revision 1)
> +    sorted_ = sorted(values)
> +    med = int(len(sorted_) / 2)
> +
> +    if len(sorted_) % 2:
> +        return sorted_[med]
> +    return (sorted_[med - 1] + sorted_[med]) * 0.5

Lets wrap this in an `int`.

::: testing/awsy/awsy/process_perf_data.py:96
(Diff revision 1)
>      total = 0
>      for checkpoint in checkpoints:
>          memory_report_path = os.path.join(data_path, checkpoint['path'])
>  
>          name_filter = checkpoint.get('name_filter', None)
>          count = checkpoint.get('count', 0)

`count` was a hack to roughly support this, we can remove that now.

Comment 3

[Kris Maglione [:kmag]]

Comment on attachment 8990611 [details]
Bug 1474208: Use median of all process values for AWSY base content.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/255670/diff/1-2/

Comment 4

[Kris Maglione [:kmag]]

Comment on attachment 8990611 [details]
Bug 1474208: Use median of all process values for AWSY base content.

https://reviewboard.mozilla.org/r/255670/#review262646

> Lets wrap this in an `int`.

Switched from `* 0.5` to `/ 2` which automatically returns an int when its LHS is an int.

Comment 5

[Eric Rahm [:erahm]]

Comment on attachment 8990611 [details]
Bug 1474208: Use median of all process values for AWSY base content.

https://reviewboard.mozilla.org/r/255670/#review262712

Comment 6

[Kris Maglione [:kmag]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8504c0085847c091aec4268699c6eeb23356c58c
Bug 1474208: Use median of all process values for AWSY base content. r=erahm

Comment 7

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/8504c0085847