785373 - Switch off peptest

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ed Morley [:emorley]]

Peptest is currently running on mozilla-central, inbound & Try, but has been hidden everywhere due to staying green even when the browser fails to start - and therefore not being very useful.

We're currently wasting the limited infra capacity running this test hidden - so we should either fix it now (and unhide) or switch it off until someone has the time.

Comment 1

[Andrew Halberstadt [:ahal]]

The decision to hide it was made by mcote and aki while I was still in school, so maybe they can comment more on the reasoning. But my understanding is that peptest was intentionally perma-greened and will purposefully remain perma-green indefinitely.

The loose reasoning was that originally peptest was designed to be a regression test suite, but due to the nature of responsiveness we were concerned it would just introduce too many random oranges. Instead we decided to track responsiveness over time and provide graphs, making it possible to track the responsiveness of a certain test. Mcote has mockup graphs here: http://people.mozilla.com/~mcote/peptest/results/#/fedora/test_contextMenu.js/sum (obviously they haven't been getting new data in a long time).

Re: wasting resources, the tests themselves take about 5 minutes, so it doesn't seem like that bad of a hit. They are also already unhidden on both Mac and Linux (Windows was hidden because of an infra exception but that is now fixed so it could also be unhidden).

Admittedly no one is really using peptest, and I am a little swamped with b2g work, but I looped in taras as the snappy team are the ones who could be potentially using it. If there is a need to get it into a good working state sooner rather than later, I can ask about freeing up some of my b2g work to focus more on peptest. I think this is a good discussion to have now.

Comment 2

[Ed Morley [:emorley]]

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> The decision to hide it was made by mcote and aki while I was still in
> school

It was made by philor (and I was in agreement), at some point after it was unhidden post bug 774817.

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> Re: wasting resources, the tests themselves take about 5 minutes, so it
> doesn't seem like that bad of a hit. 

It's more that sheriffs (and devs) have said "wait times are too high!" [1], to which one of releng's responses has been (and I paraphrase from IRC/planet posts/dev.*) "yeah but you are running lots of busted/hidden/redundant tests, so the situation is as much a product problem as it is infra... so asking us to make promises is a bit hypocritical" (latter was in response to [2]).

To try and (a) reduce infra wastage in (the relatively few) cases where we really are, and (b) be able to put the ball 100% back in releng's court (by having proof that the remaining hidden tests are still being used), I filed bug 784681 & a number of dependants, of which this is one.

If peptest is useful in its current form, then I'm not trying to take it away from you - we just need to say to releng that peptest is working+used :-)

[1] http://mozillamemes.tumblr.com/post/29065011033/the-solution-is-to-manually-coalesce-future-builds always makes me giggle :-)

[2] https://groups.google.com/d/msg/mozilla.dev.platform/nrbJE1ixkIs/y8a9ZKUYd_wJ

Comment 3

[Andrew Halberstadt [:ahal]]

No I agree, and I don't think it is useful in it's current form. We wanted to get some example tests running on tbpl so devs could start to familiarize themselves with it and try writing their own tests. To date that hasn't happened, so maybe you're right and removing it for now is the right way to go. I'll let ctalbert and taras chime in here.

Comment 4

[Andrew Halberstadt [:ahal]]

Er, s/is useful/is being used/

Comment 5

[(dormant account)]

(In reply to Andrew Halberstadt [:ahal] from comment #3)
> No I agree, and I don't think it is useful in it's current form. We wanted
> to get some example tests running on tbpl so devs could start to familiarize
> themselves with it and try writing their own tests. To date that hasn't
> happened, so maybe you're right and removing it for now is the right way to
> go. I'll let ctalbert and taras chime in here.

Yes, lets remove it until tests are tweaked to be more useful.

Comment 6

[Ed Morley [:emorley]]

(In reply to Taras Glek (:taras) from comment #5)
> Yes, lets remove it until tests are tweaked to be more useful.

Great - thanks :-)

Comment 7

[Ed Morley [:emorley]]

Attachment: Comment out peptest

Comment 8

[Ed Morley [:emorley]]

Attachment: Disable peptest

Removing the peptest block entirely, since:
bhearsum|buildduty: edmorley: can you rip it out completely instead? we should do it in a better way when we want it back

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 656060 [details] [diff] [review]
Disable peptest

Review of attachment 656060 [details] [diff] [review]:
-----------------------------------------------------------------

We'll do this in a less hacky way next time.

Comment 10

[Ed Morley [:emorley]]

Comment on attachment 656060 [details] [diff] [review]
Disable peptest

http://hg.mozilla.org/build/buildbot-configs/rev/d4d61b553447

Comment 11

[[e-mail] (:bhearsum)]

Made it to production today.