1139904 - Add mochitest browser-chrome and devtools runtime files to testing/runtimes

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Andrew Halberstadt [:ahal]]

In order to chunk things based on the test runtimes, we need to have data on the runtimes checked into the tree. I propose a simple JSON format where each file represents a single job (platform+suite specific). It would be of the form:

{ <test identifier>: <average runtime> }

I'd propose a file structure like:

testing/config/runtimes/win32-opt/mochitest-plain.json

The harness would be responsible for finding the correct runtimes.json file and passing it in to the chunking algorithm.

Comment 1

[Andrew Halberstadt [:ahal]]

Hm, the mochitest runtimes files total about 33MB for all tests. Do people think that is too much to check into the tree (keep in mind this only counts all variants of mochitest)? If that is too large, I see three options:

1) Only check-in runtime data for tests slower than a certain threshold. A threshold of 500ms drops the size down to 16MB. This approach favours slower platforms like windows where there is a much higher percentage of tests above 500ms. That might not be a bad thing as those platforms likely need normalizing the most.

2) Only checkin runtime data above a certain percentile. This scales linearly, e.g above 50th percentile is ~17MB. This also treats all platforms fairly.

3) Store all runtime data in some other repo and have a pointer to it via commit SHA in a file.

I think it would be nice to have data on every test, not just certain ones.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

What do we think we can normalize with this data?  Right now it is browser-chrome and dev-tools?  If so, what does that look like?

Even with the reduced set, we should be smart about how we do this.  

Can we assume all tests have a 500ms runtime and if it is more annotate it (I assume that is what you are proposing).  

Maybe we could have the data easily accessible in a 3rd party repo and have a summarized or hashed version checked into the tree?

If i had to choose one of the 3 above options, I would go for #2.

Comment 3

[Andrew Halberstadt [:ahal]]

(In reply to Joel Maher (:jmaher) from comment #2)
> What do we think we can normalize with this data?  Right now it is
> browser-chrome and dev-tools?  If so, what does that look like?

The 33MB would let us normalize every mochitest variant on every platform, including ones that still use "json" style manifests like android.

> Can we assume all tests have a 500ms runtime and if it is more annotate it
> (I assume that is what you are proposing).

I was going to assume that all tests below 500ms would have a runtime of 0, as they are "negligible". Or maybe we could calculate the average of all tests that aren't included and use that as the default. I'd still prefer making use of all the data if possible though.

Comment 4

[Andrew Halberstadt [:ahal]]

Jgriffin, ted, I'd like your opinions as well re: comment 1. How concerned should I be about adding 33MB to the tree?

Comment 5

[Jonathan Griffin (:jgriffin)]

If we want to continue supporting the running of tests without network access (at least for Firefox), then we'll have to keep this data in the tree in some fashion.

Do we have a sense for how the normalization would be affected as we change the minimal threshold for inclusion in this file?  E.g., would we still get fairly good normalization if 1s was the minimum threshold?  How about 5s?

What's the disadvantage of annotating the manifests themselves with this data, rather than adding a monolithic JSON file?

Comment 6

[Andrew Halberstadt [:ahal]]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> If we want to continue supporting the running of tests without network
> access (at least for Firefox), then we'll have to keep this data in the tree
> in some fashion.

We could put them on tooltool or something like that and have the slaves download them. But yes, I'd prefer if they were in-tree.

> Do we have a sense for how the normalization would be affected as we change
> the minimal threshold for inclusion in this file?  E.g., would we still get
> fairly good normalization if 1s was the minimum threshold?  How about 5s?

No, I'm not sure yet. It will be difficult to say until we can do some try pushes.

> What's the disadvantage of annotating the manifests themselves with this
> data, rather than adding a monolithic JSON file?

Mostly it will make them harder to read (keep in mind it's 1 runtime per platform per test). It's data that's intended to be read by machines, so it made sense to store it in JSON. Also it's not one monolithic file, it's one file per suite per platform, e.g:
testing/config/runtimes/win32-opt/mochitest-plain.json

I toyed with the idea of storing runtimes files next to the tests, and adding a 'runtimes' section to each mochitest.ini that points to it. But this will make both the chunking and runtimes generating algorithms more complicated and I'm not sure if it will save all that much space.

Comment 7

[Andrew Halberstadt [:ahal]]

Another option to reduce size of the runtimes files would be to average all of the platforms together. This way the runtimes would be accurate relative to one another, but not for any one platform. Then we only need to store one copy of each test instead of N copies (for N platforms).

I think this is what I'm going to land for now.. we can always change it later.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Wouldn't b2g be different than android and different than desktop?

How do we differentiate ASAN, pgo, opt, debug, e10s?

Comment 9

[Andrew Halberstadt [:ahal]]

(In reply to Joel Maher (:jmaher) from comment #8)
> Wouldn't b2g be different than android and different than desktop?
> 
> How do we differentiate ASAN, pgo, opt, debug, e10s?

Yes they would, but that's ok. This method assumes that the *relative* test runtimes are roughly the same across platforms. E.g it assumes that if test A is twice as fast as test B on linux opt, then it is also roughly twice as fast as test B on emulator debug.

Obviously this won't always be the case, but I suspect that we can still get a reasonably good approximation.

Comment 10

[(not currently active) Ted Mielczarek]

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> Hm, the mochitest runtimes files total about 33MB for all tests. Do people
> think that is too much to check into the tree (keep in mind this only counts
> all variants of mochitest)? If that is too large, I see three options:
> 
> 1) Only check-in runtime data for tests slower than a certain threshold. A
> threshold of 500ms drops the size down to 16MB. This approach favours slower
> platforms like windows where there is a much higher percentage of tests
> above 500ms. That might not be a bad thing as those platforms likely need
> normalizing the most.
> 
> 2) Only checkin runtime data above a certain percentile. This scales
> linearly, e.g above 50th percentile is ~17MB. This also treats all platforms
> fairly.

We talked about something like #1 or #2 in bug 1054247 comment 14.

> 3) Store all runtime data in some other repo and have a pointer to it via
> commit SHA in a file.

NOPE

> 
> I think it would be nice to have data on every test, not just certain ones.

Comment 11

[Andrew Halberstadt [:ahal]]

Hm, so Kyle mentioned in bug 1054247 comment 15 that we might be able to get accurate results with a very high threshold.. maybe I'll just land all tests for all platforms above the 90th percentile (3.5MB) for now. Maybe that will be good enough. If not we can revisit.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

ok, 90th percentile sounds great.  I would be surprised that the difference between opt/debug/pgo/e10s is roughly the same independent of the platform we run on.  Cool if it is!

Comment 13

[Andrew Halberstadt [:ahal]]

Attachment: MozReview Request: bz://1139904/ahal

/r/5247 - Bug 1139904 - Add initial test runtimes files for mochitest, r=jmaher

Pull down this commit:

hg pull review -r 94cedc21d29c0f1db9ea1e7011f441ee74f37c63

Comment 14

[Andrew Halberstadt [:ahal]]

Initially I uploaded this to the wrong bug and Joel replied over there. Here is his initial comment and my response.

(In reply to Joel Maher (:jmaher) from comment #3)
> Comment on attachment 8576256 [details]
> MozReview Request: bz://1142050/ahal
> 
> https://reviewboard.mozilla.org/r/5221/#review4229
> 
> overall this looks great.  I was under the impression that we had a common
> multiplier for all platforms for a given test between opt/debug?  also how
> does pgo, e10s, ASAN fit into this?

You and ted seemed to have reservations about averaging the platforms together, and on second thought I agreed.. so I thought I'd just add the top 90th percentile instead, which keeps the files down to a relatively small size. We can always change how the data is generated later after we get an idea of how well this works out.

This data was generated using a one off script, it's mostly so we can start playing around with runtime chunking. Eventually we'll have to write a tool that generates the data using Kyle's active data project as a source, which should include PGO and ASAN. There are already some e10s suites included in here. This data should be more than enough to get us started playing around on try.

Comment 15

[Andrew Halberstadt [:ahal]]

Comment on attachment 8576639 [details]
MozReview Request: bz://1139904/ahal

/r/5247 - Bug 1139904 - Add initial test runtimes files for mochitest, r=jmaher

Pull down this commit:

hg pull review -r 5708c8cfb400de4e09315c721ebf7a9ff6c4a4b0

Comment 16

[Andrew Halberstadt [:ahal]]

This latest version moves the runtimes files from testing/mochitest/runtimes to testing/runtimes.. I figured it would be easier to generate them if they were all in the same directory (once we have more than just mochitest)

Comment 17

[Andrew Halberstadt [:ahal]]

Comment on attachment 8576639 [details]
MozReview Request: bz://1139904/ahal

/r/5247 - Bug 1139904 - Add initial test runtime files for mochitest browser-chrome and devtools, r=jmaher

Pull down this commit:

hg pull review -r 0d65ed70715c5c89ff825a4bef30902938e1cd86

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8576639 [details]
MozReview Request: bz://1139904/ahal

https://reviewboard.mozilla.org/r/5245/#review4275

Ship It!

Comment 19

[Andrew Halberstadt [:ahal]]

Comment on attachment 8576639 [details]
MozReview Request: bz://1139904/ahal

/r/5247 - Bug 1139904 - Add initial test runtime files for mochitest browser-chrome and devtools, r=jmaher

Pull down this commit:

hg pull review -r 42964dc215bd0cdf8c2c98cf80ed7e36439268db

Comment 20

[Andrew Halberstadt [:ahal]]

Comment on attachment 8576639 [details]
MozReview Request: bz://1139904/ahal

Talked to Joel on irc, this version adds a README and a missing devtools-chrome file on win64-opt. Carrying r+ forward.

Comment 21

[Andrew Halberstadt [:ahal]]

Here's a try run out of an abundance of caution:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=42feaf5a3d4c

Landed:
https://hg.mozilla.org/integration/mozilla-inbound/rev/106237ea5c7e

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/106237ea5c7e

Comment 24

[Andrew Halberstadt [:ahal]]

Attachment: MozReview Request: Bug 1139904 - Add initial test runtime files for mochitest browser-chrome and devtools, r=jmaher