821377 - crashtests should be a single chunk

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

now that the crashtests can run to completion, we need to turn them all on.  In looking at the total runtime, we do 6-10 minutes of testing/chunk, and 15 minutes of setup.  I would like to run these in a single chunk for mozilla-central based branches.

Worse case scenario we would run all crashtests in 2 chunks on central based branches.

Comment 1

[Justin Wood (:Callek)]

I'm tossing back into the pool, the timeline I forsaw for this doesn't line up with the work involved when needing to do this for a subset of all branches at the outset.

Comment 2

[Phil Ringnalda (:philor)]

Wouldn't a prerequisite for running all crashtests in one hunk be to first fix bug 752373 and turn crashtest-1, which we haven't run for a few days shy of a year because nobody was willing to fix it, back on?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

when this bug was filed and for 3 months after it was filed the crashtests would all pass consistently on tegra and panda.  There is 1 new test case as of March 29th (bug 852838), which causes a failure.  Once that is addressed we can run these.

run times:
~9:28 panda
~10:15 tegra

Add setup time of up to 20 minutes each and we have <30 minutes for a single crashtest job to complete.

Comment 4

[Phil Ringnalda (:philor)]

I think the disconnect is that you are saying "when I run crashtests on my tegra and my panda so that I'm running the stuff in crashtest-1" and I'm hearing "when I look at tbpl, the crashtests that are run there."

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

you will need to remember that I always request tests to be turned on after running them 10 times in a row on my local machines.  This is how the tests got turned on in the first place.  Whilst there might be something I am overlooking in C1, I do recall C1 being flaky on my local tegra a year ago, it seems much more stable now.

Comment 6

[Phil Ringnalda (:philor)]

Well, as it turns out, the delay in fixing this makes it much easier - since the trunk when you filed it was Fx 20, and Fx 20 is now on mozilla-release, and we don't run Android crashtests anywhere below mozilla-release, there's no need to only run it on the trunk first. We can just add a new suite next to crashtest-2 and crashtest-3, named crashtest, reconfig, if all hell breaks loose somewhere hide it, if not kill crashtest-2 and crashtest-3 on the next reconfig.

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: patch

to reflect request in comment #6

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 744633 [details] [diff] [review]
patch

Review of attachment 744633 [details] [diff] [review]:
-----------------------------------------------------------------

I believe this looks right.

Comment 9

[Kim Moir [:kmoir] ET]

Comment on attachment 744633 [details] [diff] [review]
patch

Once we see that the single crashtest works we can removed the crashtest-1 and crashtest-2 as philor suggests.  I tested it on my dev-master and it seemed fine.

Comment 10

[Aki Sasaki (not active)]

This is in production.

Comment 11

[Phil Ringnalda (:philor)]

Callek hid it on Try (in an odd reversal of our usual style of hiding very broken suites elsewhere and leaving them visible on Try); I hid it on mozilla-beta, though I'm hoping that https://hg.mozilla.org/releases/mozilla-beta/rev/98ddc0509f20 will green it up there.

Comment 12

[Phil Ringnalda (:philor)]

Green and unhidden on beta; it'll probably be busted on release, but won't be there long enough for anyone to care. Makes for an interesting situation - bug 868152 still hopes we can disable our way to victory on the trunk, but as of now we could shut off crashtest-2 and crashtest-3 on everything *but* the trunk, unlike the original plan where we would only be making the switch *on* the trunk.

Unhidden on Try since as of the tip of m-c, at least it's not permaorange, quite.

Comment 13

[Phil Ringnalda (:philor)]

The second half, shutting off C2/C3, moved to bug 869748, so this is done. Thanks!