Please schedule jit-tests on Try

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: dminor, Assigned: dminor)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

4 years ago
Jit-test failures on Cedar seem to be down to intermittents rather than automation problems. Scheduling these on Try would make it easier for developers to track down and resolve these intermittents. Besides Cedar, the only place in automation these run at the moment is part of make-check, which isn't the best for debugging.

The last few builds on Cedar have not run Android Panda jit-tests successfully - they seem to retry endlessly. Because of this, I think we should only schedule the tests for the desktop platforms until the panda problems are sorted out.

The visibility for these should be hidden for now.
(Assignee)

Comment 1

4 years ago
The retries appear to be due to a single test timing out: see bug 931882.

Comment 2

4 years ago
Created attachment 823459 [details] [diff] [review]
bug931874.patch
Assignee: nobody → kmoir

Updated

4 years ago
Attachment #823459 - Flags: review?(bugspam.Callek)

Updated

4 years ago
Blocks: 932238

Updated

4 years ago
Attachment #823459 - Flags: review?(bugspam.Callek) → review+
https://hg.mozilla.org/build/buildbot-configs/rev/071cfc93d1c7
That patch appears to enable them on every platform, and since bug 931882 isn't resolved, that means the next reconfig is going to enable infinite retries on Try, which seems like a Very Bad Thing.
Backed out in https://hg.mozilla.org/build/buildbot-configs/rev/4178f9428f32 on the assumption that Very Bad Things are not Things That We Want.
in production
(In reply to Phil Ringnalda (:philor) from comment #4)
> That patch appears to enable them on every platform, and since bug 931882
> isn't resolved, that means the next reconfig is going to enable infinite
> retries on Try, which seems like a Very Bad Thing.

Where did you get the knowledge/idea this was all platforms? I didn't verify with a dump but afaik its not affecting b2g-only platforms nor testing android/fennec platforms.

I'd love a piece of data here, since this is a bug that ateam was driving to get in sooner than later.
Flags: needinfo?(philringnalda)

Updated

4 years ago
Assignee: kmoir → bugspam.Callek
(Assignee)

Comment 8

4 years ago
These should not be run on Android until either bug 931882 or bug 882179 is resolved. Otherwise, the jit tests will retry indefinitely.
Flags: needinfo?(philringnalda)
(In reply to Dan Minor [:dminor] from comment #8)
> These should not be run on Android until either bug 931882 or bug 882179 is
> resolved. Otherwise, the jit tests will retry indefinitely.

Thats just it, based on my understanding of the patch this was not turning it on for android. Hence my needinfo to philor for info on why he thought it was (because I can't do a dump to verify locally for a few hours)
Flags: needinfo?(philringnalda)
On Cedar, we are running jittests on Android. They infinitely retry there, until someone manually kills them.

This patch added what looks to me like exactly the same things that are running on Cedar, by just adding in Try in the places that were formerly only Cedar, without any attempt at removing Android.

Curiously enough, post-reconfig and despite my backout, we now *are* running jittests on Try on Android, and only on Android.
Flags: needinfo?(philringnalda)
(In reply to Phil Ringnalda (:philor) from comment #10)
> Curiously enough, post-reconfig and despite my backout, we now *are* running
> jittests on Try on Android, and only on Android.

Ooooooooo FFS I see why this patch doesn't do it, your backout didn't fix it, but indeed they are running on try,

Unrelated  patch did this and I didn't notice:
http://hg.mozilla.org/build/buildbot-configs/rev/5e7078dcbb58

note the mobile_config.py piece, _that_ needs to be backed out and is what enables mobile jit-tests anywhere, not this patch which you backed out.
Oh. Yeah, that patch is busted anyway, since it only enabled Cpp on debug Mac and Windows, not on opt and not on any Linux.
(Assignee)

Comment 13

4 years ago
Relanding this as it seems this patch worked as intended.

https://hg.mozilla.org/build/buildbot-configs/rev/f3fe5fcc354b

Updated

4 years ago
Depends on: 942200
something[s] here made it to production
Dan, thanks for working on this! However, I noticed we run jit_test.py without --tbpl. This flag is necessary so that we run all tests with a number of interesting flags like --ion-eager.

Note that "make check" and the standalone SM builds use this flag as well.
Flags: needinfo?(dminor)
(Assignee)

Comment 16

4 years ago
Thanks! Good catch, I've filed Bug 943411 to resolve this.
Flags: needinfo?(dminor)
(Assignee)

Comment 17

4 years ago
I've asked for these to be hidden for now. In case anyone is unfamiliar with the syntax, appending &showall=1 to the tbpl url will show all tests.
(Assignee)

Comment 18

4 years ago
Created attachment 8338779 [details] [diff] [review]
Part 2, enable jit-tests for Android on Try

Now that bug 931882 is fixed, we can enable these on try for Android as well.
Assignee: bugspam.Callek → dminor
Status: NEW → ASSIGNED
Attachment #8338779 - Flags: review?(jgriffin)
Attachment #8338779 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 19

4 years ago
Thanks, pushed to https://hg.mozilla.org/build/buildbot-configs/rev/dc77a8c893dc
in production
(Assignee)

Updated

4 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Fixing bug 1080224 got me wondering if and when we were planning to ever get these green? Otherwise, we should probably just turn them off instead of pretending like we somehow care about them, wasting resources along the way.
Flags: needinfo?(dminor)
(Assignee)

Comment 22

3 years ago
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #21)
> Fixing bug 1080224 got me wondering if and when we were planning to ever get
> these green? Otherwise, we should probably just turn them off instead of
> pretending like we somehow care about them, wasting resources along the way.

Looking at Cedar, these actually had a few green runs recently (on opt). Before that, they were permanently orange for as long as I can remember - I believe the parallel suite failed more often than it succeeded.

I believe the js group is currently running these tests manually, which implies they work on phones, but not on the pandaboards. Terrence, are you still interested in running these in automation? If so, I'll file bugs for any current failures, but we would need someone from the js group to investigate them.
Flags: needinfo?(dminor) → needinfo?(terrence)
Yes, we absolutely want to keep running these. Fortunately, we have plans to make the parallel tests much more stable in the near future, so this should be viable once that lands. I'll keep myself flagged and we can kick off an initiative (hopefully sometime in December) to get these permanently enabled.
https://tbpl.mozilla.org/?tree=Try&rev=e6f9f7154b36&showall=1
(Assignee)

Comment 25

3 years ago
Filed bug 1098508 to get these fixed and scheduled elsewhere.
Flags: needinfo?(terrence)
Thanks, Dan!
You need to log in before you can comment on or make changes to this bug.