Closed Bug 823660 Opened 10 years ago Closed 10 years ago

for Android NoIon Only run JSReftests and builds

Categories

(Release Engineering :: General, defect)

ARM
Android
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: joduinn, Assigned: Callek)

Details

Attachments

(1 file, 4 obsolete files)

per email w/danders:

Because we are now running b2g builds and tests, the no-ionmonkey builds & tests are now obsolete. This bug is to stop running no-ion, and remove from the configs, all traces of these no-ion builds and tests.
OS: Mac OS X → Android
Hardware: x86 → ARM
Assignee: nobody → bugspam.Callek
Attached patch [configs] v1 - rip it out (obsolete) — Splinter Review
Attachment #694531 - Flags: review?(aki)
Blocks: 823676
Attached patch [custom] v1 (obsolete) — Splinter Review
Attachment #694532 - Flags: review?(aki)
Attachment #694535 - Flags: review?(aki)
Attached patch [puppet-manifests] v1 (obsolete) — Splinter Review
This patch should land last, *after* reconfig for above patches.
Attachment #694536 - Flags: review?(aki)
CCing massimo because I'm touching the old-puppet files for bm's so its not forgotten when he goes to get ready to land the new manifests.
Does this mean that we would only be testing the non-ion js engine with the emulator?
I care so little about testing a JIT in an emulator that I would not even bother.  I think there's a misunderstanding that the b2g tests are running on real hardware.

If we turn these off, we'll be shipping a js engine that's not tested to the degree it is on any other platform.  Please no?
(In reply to Chris Jones [:cjones] [:warhammer] from comment #7)
> I think there's a misunderstanding that the b2g tests are running
> on real hardware.

To be clear, they're not running on real HW: they're running in qemu on linux-x86 hosts.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #7)
> I care so little about testing a JIT in an emulator that I would not even
> bother.  I think there's a misunderstanding that the b2g tests are running
> on real hardware.
> 
> If we turn these off, we'll be shipping a js engine that's not tested to the
> degree it is on any other platform.  Please no?

Is qemu expected to be unreliable with JITs? For the record, we've found ARM-only JIT bugs that occur on non-Tegra hardware (due to using undefined instruction behavior) so it seems hard to get really good JIT test coverage without a broad range of CPU implementations.

Anyway, if B2G considers qemu unacceptable for JIT coverage, we might as well keep the hardware testing.
If the emulator isn't sufficient, could we at least stop all NoIon tests apart from the jsreftests?
Definitely, if the JS folks consider those sufficient.
(In reply to David Anderson [:dvander] from comment #9)
> (In reply to Chris Jones [:cjones] [:warhammer] from comment #7)
> > I care so little about testing a JIT in an emulator that I would not even
> > bother.  I think there's a misunderstanding that the b2g tests are running
> > on real hardware.
> > 
> > If we turn these off, we'll be shipping a js engine that's not tested to the
> > degree it is on any other platform.  Please no?
> 
> Is qemu expected to be unreliable with JITs? For the record, we've found
> ARM-only JIT bugs that occur on non-Tegra hardware (due to using undefined
> instruction behavior) so it seems hard to get really good JIT test coverage
> without a broad range of CPU implementations.

I don't expect it to be unreliable, I just expect it to not be a useful test target since it's on a different planet implementation-wise from real HW.

It would be ideal if we could test on the target HW, but we can't do that at scale unfortunately.  However, not-target real HW >>> qemu.
Put another way, would you be comfortable shipping an x86 JIT for desktop Firefox after it's only been tested inside a software emulator running on MIPS, say?
Yeah, I would say that's not ideal :) 

re: jsreftests, yeah that should be sufficient.
Even just switching off everything but jsreftests would save ~240 out of 330mins total test runtime, which would be great from a tegra load POV :-)
cjones, dvander: 

After reading the last few comments, its unclear to me whats being asked of us. Can you post here a concise summary of which builds/tests you both agree we should keep running, and which to stop running?
Please run the android-noion jsreftests on real hardware.
(In reply to John O'Duinn [:joduinn] from comment #16)
> cjones, dvander: 
> 
> After reading the last few comments, its unclear to me whats being asked of
> us. Can you post here a concise summary of which builds/tests you both agree
> we should keep running, and which to stop running?

(In reply to Chris Jones [:cjones] [:warhammer] from comment #17)
> Please run the android-noion jsreftests on real hardware.

cjones, dvander: paraphrasing to make sure I have this correct. The ask is to have Releng continue running some tests, and drop some tests, as follows:

1) continue generating no-ion build, and running jsreftest on hardware. On tbpl.m.o, for "Android 2.2 NoIon opt", this looks like "B R(J1 J2 J3)".
2) stop running mochitests on no-ion builds. On tbpl, for "Android 2.2 NoIon opt", this means removing M(1 2 3 4 5 6 7 8 rc)".

If this is correct, please ack. If I've missed something, please clarify.
Summary: Stop running all Android NoIon builds and tests → for Android NoIon Only run JSReftests and builds
Attachment #694531 - Attachment is obsolete: true
Attachment #694532 - Attachment is obsolete: true
Attachment #694535 - Attachment is obsolete: true
Attachment #694536 - Attachment is obsolete: true
Attachment #694531 - Flags: review?(aki)
Attachment #694532 - Flags: review?(aki)
Attachment #694535 - Flags: review?(aki)
Attachment #694536 - Flags: review?(aki)
Attachment #696211 - Flags: review?(bhearsum)
Attachment #696211 - Flags: review?(bhearsum) → review+
in production
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
No longer blocks: 823676
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.