851248 - please turn off tp5n for mozilla-central and sibling branches, leave tp5o running.

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

After looking at our new tp5o data, it is evident that we have a solid set of data to work with and can move forward with much more accurate regression detection and reporting.  Please turn off tp5n.

For reference, tp5o runs 10-13 minutes faster than tp5n, this will save on machine time!

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: default to tp5o, but for all non m-c branches use tpn (1.0)

Comment 2

[Armen [:armenzg]]

Comment on attachment 725437 [details] [diff] [review]
default to tp5o, but for all non m-c branches use tpn (1.0)

Looks good. It follows the train and all.

Thanks jmaher!

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

landed:
http://hg.mozilla.org/build/buildbot-configs/rev/0db18af2fa0b

Comment 4

[Chris Cooper [:coop] (he/him)]

In production now.

Comment 5

[Phil Ringnalda (:philor)]

Don't forget the part where you update trychooser.

Comment 6

[Armen [:armenzg]]

Attachment: add tp5o to the trychooser

Comment 7

[Phil Ringnalda (:philor)]

I'd say "r=me as long as you also remove tpn, since it's gone," but checking that it was really gone made me notice that tp5o on Try is as close to utterly worthless as you can get - it doesn't post to graphserver, it doesn't TinderboxPrint results, it only gives a Datazilla URL that says absolutely nothing other than "no metrics data available," so I think the right thing to do here is not to change trychooser, but is instead to back out removing tp5n until tp5o can actually serve as an adequate replacement for it.

Comment 8

[Jeff Hammel]

I forget our story for try (both short term and long term). We should have one.  :jeads? :jmaher?

Comment 9

[Jonathan Eads ( :jeads )]

Story for Try in Datazilla
--------------------------
We don't have a way to identify the repository to compare data to for a given Try push when we receive talos data in datazilla. You cannot compute a t-test without two sets of replicates, without the repository to compare to, we only have one set of replicates for the Try push. As soon as we can resolve this we can display data for Try in datazilla.

Any ideas? Could we choose a default repository to compare to until we have a better solution?

Comment 10

[Jeff Hammel]

*Ideally* I think having a controllable parameter for try syntax would be the way to go.  That said, I think we're chicken+egged here: without datazilla proving its worth, supporting this will be poo-pooed, and without this....

That said, I think we could get away with mozilla-central/inbound being the default with a good success rate.  When it fails it could really fail, but in these latent times I think as long as we inform people of this....somehow, that is fair.  Even with the ability to choose trees on try, I would still want central/inbound as the default, if for no other reason than everyone would complain about too much typing were it not.

Comment 11

[Armen [:armenzg]]

Attachment: add tp5o to the trychooser and clean up

The patch is awaiting decision on the way to go.

tp5o is broken on try.
Can I backout the original patch until we find out a way to fix it?
Or can we show tp5o results on try and allow people to compare manually if they need to?

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

I have been trying to test on try server my patch to post tp5o results to graph server.  I am really ready to land it, but I wanted to make sure I didn't break anything.  I am fine backing patches out or whatever we need to do.

Comment 13

[Ed Morley [:emorley]]

(In reply to Jonathan Eads ( :jeads ) from comment #9)
> Any ideas? Could we choose a default repository to compare to until we have
> a better solution?

Default to mozilla-central or mozilla-inbound; perhaps the latter, since it has more replicates per day and thus presumably a more reliable trend mean?

Comment 14

[Armen [:armenzg]]

Deployed to trychooser.