1393068 - AWFY - determine which repositories should be tested.

Status: RESOLVED WONTFIX

(Testing Graveyard :: AWFY, enhancement)

Description

[Bob Clary [:bc] (inactive)]

In bug 1392123 there is a mention of some desire for mozilla-central and autoland testing in AWFY. We currently only test mozilla-inbound and mozilla-beta.

Which repos should we be testing?

I would expect we should run autoland every hour since this is where the JS team is landing stuff.

I don't see much value and possibly too much load if we enabled mozilla-central and/or mozilla-inbound.

Ben, do you think that if we wanted to swap autoland for mozilla-inbound would changing the scripts and database entries to reference autoland instead of mozilla-inbound be sufficient? I'd rather not duplicate the entire mozilla-inbound entries everywhere.

Comment 1

[Benjamin Bouvier [:bbouvier] (inactive)]

I think most people in the JS team use mozilla-inbound, but I could be mistaken because I only think at the people I closely work with (?).

Agreed that having central and inbound would probably add too much load for too little value, since the two repositories seem to be regularly synced. But then, the same question maybe applies to autoland too: how often is autoland merged into inbound and conversely?

Running on inbound allows us to spot regressions quickly. A few machines already run speedometer2 on inbound: the quantum ref machine, and the windows machine (that's not a Mac Mini). So one nice trade-off would be to use the latter for inbound, and other repositories (including autoland) on the quantum ref machine. This way, we can still have both the SM2 score on the QF machine, and spot regressions with the Windows machine. In this case, it'd be ok to increase load on the QF machine. Jan, as somebody interested by both cases, does it make sense to you?

> Ben, do you think that if we wanted to swap autoland for mozilla-inbound would changing the scripts and database entries to reference autoland instead of mozilla-inbound be sufficient? I'd rather not duplicate the entire mozilla-inbound entries everywhere.

In general, I think all the non-QF benchmarking tasks should stay on inbound. So it'd be better if we'd add new entries instead of changing them.

I haven't ever done this, so I can only guess here. I think changing the task commands and adding a new awfy_mode and/or awfy_vendor should be enough.

Comment 2

[Bob Clary [:bc] (inactive)]

Ben, thank you!

I'm happy to stay with inbound for now. I'd rather not do something just for the sake of doing it. If the JS team is ok with inbound the way it is now, please WONTFIX this.

Comment 3

[Bobby Holley (:bholley)]

The basic issue is that the JS team uses inbound, and the stylo team uses autoland. Speedometer in general is much less sensitive to our work than to the JS team's, so I wouldn't suggest switching from inbound to autoland, I was merely suggesting doing the stylo-mode runs on autoland. If that's a hassle or wasteful, it's probably not worth it, and we can wait for our changes to be merged over to autoland.

Comment 4

[Benjamin Bouvier [:bbouvier] (inactive)]

It seems that thanks to Armen's recent work, we can easily change download.py to support autoland. In this case, running Speedometer2 on autoland would be easy; a few awfy_vendor/awfy_mode insertions and tweaks to the url_creator scripts should be sufficient here.

Bob, what do you think?

Comment 5

[Benjamin Bouvier [:bbouvier] (inactive)]

(It appears having the AWFY scripts supporting autoland is trivial; so it's just a matter of inserting the right vendor/mode lines indeed; see also https://github.com/mozilla/arewefastyet/pull/166 )

Comment 6

[Bob Clary [:bc] (inactive)]

I'd like to see how we do in bug 1392123 before committing to more additions. I believe we could add speedometer on autoland with and without stylo if we adjusted the inbound schedules to reduce the load.

Comment 7

[Jan de Mooij [:jandem]]

Sounds good to me. The quantum reference machine already has mozilla-beta data so it seems we could use a similar strategy for Stylo/autoland.

Maybe we should test stylo on machine 35 as well, the data from that machine is sometimes more useful than the reference laptop.

Comment 8

[Chris Peterson [:cpeterson]]

(In reply to Bob Clary [:bc:] from comment #6)
> I'd like to see how we do in bug 1392123 before committing to more
> additions. I believe we could add speedometer on autoland with and without
> stylo if we adjusted the inbound schedules to reduce the load.

Nearly all Stylo changes land on autoland, not inbound. So if you add Speedometer on autoland with Stylo, then you can stop running on inbound with Stylo.

Also, the Stylo code in the mozilla-beta repo (Beta 56) is outdated, so there is not much value in running Speedometer on Beta with Stylo at this time. Testing Stylo on Beta will become interesting after 57 rides from Nightly to Beta.

Comment 9

[Bob Clary [:bc] (inactive)]

awfy is gone -> wontfix.