Closed Bug 785373 Opened 9 years ago Closed 9 years ago

Switch off peptest

Categories

(Release Engineering :: General, defect)

defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Assigned: emorley)

References

Details

(Whiteboard: [capacity])

Attachments

(1 file, 1 obsolete file)

Peptest is currently running on mozilla-central, inbound & Try, but has been hidden everywhere due to staying green even when the browser fails to start - and therefore not being very useful.

We're currently wasting the limited infra capacity running this test hidden - so we should either fix it now (and unhide) or switch it off until someone has the time.
The decision to hide it was made by mcote and aki while I was still in school, so maybe they can comment more on the reasoning. But my understanding is that peptest was intentionally perma-greened and will purposefully remain perma-green indefinitely.

The loose reasoning was that originally peptest was designed to be a regression test suite, but due to the nature of responsiveness we were concerned it would just introduce too many random oranges. Instead we decided to track responsiveness over time and provide graphs, making it possible to track the responsiveness of a certain test. Mcote has mockup graphs here: http://people.mozilla.com/~mcote/peptest/results/#/fedora/test_contextMenu.js/sum (obviously they haven't been getting new data in a long time).

Re: wasting resources, the tests themselves take about 5 minutes, so it doesn't seem like that bad of a hit. They are also already unhidden on both Mac and Linux (Windows was hidden because of an infra exception but that is now fixed so it could also be unhidden).

Admittedly no one is really using peptest, and I am a little swamped with b2g work, but I looped in taras as the snappy team are the ones who could be potentially using it. If there is a need to get it into a good working state sooner rather than later, I can ask about freeing up some of my b2g work to focus more on peptest. I think this is a good discussion to have now.
(In reply to Andrew Halberstadt [:ahal] from comment #1)
> The decision to hide it was made by mcote and aki while I was still in
> school

It was made by philor (and I was in agreement), at some point after it was unhidden post bug 774817.

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> Re: wasting resources, the tests themselves take about 5 minutes, so it
> doesn't seem like that bad of a hit. 

It's more that sheriffs (and devs) have said "wait times are too high!" [1], to which one of releng's responses has been (and I paraphrase from IRC/planet posts/dev.*) "yeah but you are running lots of busted/hidden/redundant tests, so the situation is as much a product problem as it is infra... so asking us to make promises is a bit hypocritical" (latter was in response to [2]).

To try and (a) reduce infra wastage in (the relatively few) cases where we really are, and (b) be able to put the ball 100% back in releng's court (by having proof that the remaining hidden tests are still being used), I filed bug 784681 & a number of dependants, of which this is one.

If peptest is useful in its current form, then I'm not trying to take it away from you - we just need to say to releng that peptest is working+used :-)

[1] http://mozillamemes.tumblr.com/post/29065011033/the-solution-is-to-manually-coalesce-future-builds always makes me giggle :-)

[2] https://groups.google.com/d/msg/mozilla.dev.platform/nrbJE1ixkIs/y8a9ZKUYd_wJ
No I agree, and I don't think it is useful in it's current form. We wanted to get some example tests running on tbpl so devs could start to familiarize themselves with it and try writing their own tests. To date that hasn't happened, so maybe you're right and removing it for now is the right way to go. I'll let ctalbert and taras chime in here.
Er, s/is useful/is being used/
(In reply to Andrew Halberstadt [:ahal] from comment #3)
> No I agree, and I don't think it is useful in it's current form. We wanted
> to get some example tests running on tbpl so devs could start to familiarize
> themselves with it and try writing their own tests. To date that hasn't
> happened, so maybe you're right and removing it for now is the right way to
> go. I'll let ctalbert and taras chime in here.

Yes, lets remove it until tests are tweaked to be more useful.
(In reply to Taras Glek (:taras) from comment #5)
> Yes, lets remove it until tests are tweaked to be more useful.

Great - thanks :-)
Component: Peptest → Release Engineering
Product: Testing → mozilla.org
Summary: Fix peptest so it doesn't pass even if the browser fails to start (or else switch off for now) → Switch off peptest
Whiteboard: [capacity]
Version: Trunk → other
Attached patch Comment out peptest (obsolete) — Splinter Review
Assignee: nobody → bmo
Status: NEW → ASSIGNED
Attached patch Disable peptestSplinter Review
Removing the peptest block entirely, since:
bhearsum|buildduty: edmorley: can you rip it out completely instead? we should do it in a better way when we want it back
Attachment #656055 - Attachment is obsolete: true
Attachment #656060 - Flags: review?(bhearsum)
Comment on attachment 656060 [details] [diff] [review]
Disable peptest

Review of attachment 656060 [details] [diff] [review]:
-----------------------------------------------------------------

We'll do this in a less hacky way next time.
Attachment #656060 - Flags: review?(bhearsum) → review+
Made it to production today.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.