1356122 - Run Stylo tests in sequential mode

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Chris Peterson [:cpeterson]]

Kim, we'd like to add a new "linux64-stylo-sequential" platform to run some more Stylo tests:

Stylo tests can run in either parallel or sequential mode (primarily for single-core machines). In bug 1318187, we will make parallel mode the default for all Stylo tests on inbound, autoland, and central.

To make sure we're still testing sequential mode, we'd like to create a new "linux64-stylo-sequential" platform that only runs on central and try. linux64-stylo-sequential would run reftests, mochitests, and crashtests for both debug and opt builds. We don't need to run sequential reftests on non-e10s.

e10s debug reftests
e10s debug mochitests
e10s debug crashtests

e10s opt reftests
e10s opt mochitests
e10s opt crashtests

non-e10s debug mochitests
non-e10s debug crashtests

non-e10s opt mochitests
non-e10s opt crashtests

Comment 1

[Kim Moir [:kmoir] ET]

I have a few questions for you

So my understanding is that you are asking for a new standalone linux64-stylo-sequential build with the tests listed above.  Is there a mozconfig for this build?  I didn't see one in tree.  I assume when you say try, you meant that the build can run on try but not by default?

Also, I'm going to check on the state of our AWS budget these days.

Comment 2

[Bobby Holley (:bholley)]

(In reply to Kim Moir [:kmoir] from comment #1)
> I have a few questions for you
> 
> So my understanding is that you are asking for a new standalone
> linux64-stylo-sequential build with the tests listed above.  Is there a
> mozconfig for this build?  I didn't see one in tree.

I don't think there is - it would be a normal stylo build with an environmental variable set. Nathan Froyd knows the details there.

>  I assume when you say
> try, you meant that the build can run on try but not by default?

That should be fine.

> 
> Also, I'm going to check on the state of our AWS budget these days.

Since these only run on m-c, they should be effectively budget-neutral.

Comment 3

[Nathan Froyd [:froydnj]]

(In reply to Bobby Holley (:bholley) (busy with Stylo) from comment #2)
> (In reply to Kim Moir [:kmoir] from comment #1)
> > I have a few questions for you
> > 
> > So my understanding is that you are asking for a new standalone
> > linux64-stylo-sequential build with the tests listed above.  Is there a
> > mozconfig for this build?  I didn't see one in tree.
> 
> I don't think there is - it would be a normal stylo build with an
> environmental variable set. Nathan Froyd knows the details there.

This platform can use the regular stylo mozconfig, and most (all?) of the config file bits should just be copy-and-pastes of the linux64-stylo config file bits.  Once all the necessary config file bits are in place, we'd just need to modify the taskcluster transform that's going to land as part of bug 1318187.  It looks like:

@transforms.add
def parallel_stylo_tests(config, tests):
    """Ensure that any stylo tests running with e10s enabled also test
    parallel traversal in the style system."""

    for test in tests:
        if not test['test-platform'].startswith('linux64-stylo/'):
            yield test
            continue

        e10s = test['e10s']
        # We should have already handled 'both' in an earlier transform.
        assert e10s != 'both'
        if not e10s:
            yield test
            continue

        test['mozharness'].setdefault('extra-options', [])\
                          .append('--parallel-stylo-traversal')
        yield test

as that bug is trying to run stylo tests in parallel on e10s tests, and sequentially on non-e10s.  The transform would then add the appropriate mozharness option if the test was linux64-stylo only; linux64-stylo-sequential tests would be passed through unmodified.

Or we can sequence things such that the linux64-stylo-sequential configs are stood up first, and then bug 1318187 flips the linux64-stylo tests to use parallel stylo.  I'm fine either way.

Comment 4

[Kim Moir [:kmoir] ET]

garndt: I talked to catlee and he suggested talking to you about this in light of the taskcluster AWS budget since this is a request to add an additional stylo build + tests on m-c and by request on try. Is there an issue with enabling these given current budget constraints?

Comment 5

[Gregory Szorc [:gps]]

Per discussion, we don't need a whole new platform for this: it is just a variation of how the tests are executed.

Also, since we're only talking tests on Linux (for now), cost should not be an issue, as the cost of a Linux test worker is literally fractions of a penny per hour and the total cost for these extra tasks will be just a few cents per job. We'll spend more money discussing it here and implementing it than it will cost in CI in the next several months. If we're really that budget constrained, we have far bigger things to worry about.

Comment 6

[Greg Arndt [:garndt]]

gps does have a point that adding these suites do not add much to the cost per push.  The worker types used for those suites range from $0.01/hr to $0.04/hr on average.  Quick calculations show it might be < 10 cents per push unless I misplaced a zero.

For things like this, there really is no concern about adding them or not.  It is good to know when we do add more jobs because over time 10 cent additions do add up if you do it a 100 times.

Comment 7

[Chris Peterson [:cpeterson]]

Kim, do you need any additional info to run the tests in Stylo sequential mode using the existing linux64-stylo builds? Based on gps's comment 5, I don't think we need a new linux64-stylo-sequential mozconfig or build.

Comment 8

[Kim Moir [:kmoir] ET]

I think so.  Are changes needed in treeherder so the new tests appear correctly?

Comment 9

[Chris Peterson [:cpeterson]]

(In reply to Kim Moir [:kmoir] from comment #8)
> I think so.  Are changes needed in treeherder so the new tests appear
> correctly?

Sorry, I don't know. I assume the new tests would need a new test name abbreviation on Treeherder, but I don't know if both the tests and Treeherder need to know about the test name abbreviations.

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: wip patches

Comment 11

[Kim Moir [:kmoir] ET]

https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&selectedJob=96687608

Comment 12

[Kim Moir [:kmoir] ET]

Attachment: bug1356122-2.patch

Comment 13

[Kim Moir [:kmoir] ET]

Attachment: bug1356122-3.patch

fixes linting errors in 

https://treeherder.mozilla.org/#/jobs?repo=try&revision=86b69ce91bca1e0a8cb5197ca84e5e0f27264ecf&selectedJob=96966850

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c21bd752dc66
Run Stylo tests in sequential mode r=aki

Comment 15

[Iris Hsiao [:ihsiao]]

https://hg.mozilla.org/mozilla-central/rev/c21bd752dc66