Last Comment Bug 791103 - ARMv6 tests on ARMv7 for Beta 16
: ARMv6 tests on ARMv7 for Beta 16
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: All All
: -- normal (vote)
: ---
Assigned To: Justin Wood (:Callek)
:
Mentors:
Depends on: 772928 790624 790630 792300
Blocks: 785991
  Show dependency treegraph
 
Reported: 2012-09-13 14:52 PDT by Alex Keybl [:akeybl]
Modified: 2013-08-12 21:54 PDT (History)
8 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
+
fixed
+
fixed


Attachments
[configs] do-it (932 bytes, patch)
2012-09-13 15:42 PDT, Justin Wood (:Callek)
aki: review+
bugspam.Callek: checked‑in+
Details | Diff | Review
run all unit tests on armv6 (1.52 KB, patch)
2012-09-14 12:30 PDT, Aki Sasaki [:aki]
bugspam.Callek: review+
aki: checked‑in+
Details | Diff | Review

Description Alex Keybl [:akeybl] 2012-09-13 14:52:09 PDT
Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we enable all tests on Aurora/Beta, at some reasonable cadence? Thanks!
Comment 1 Justin Wood (:Callek) 2012-09-13 15:42:51 PDT
Created attachment 661024 [details] [diff] [review]
[configs] do-it

This adds reftests to aurora/beta as of today (aki should be doing a reconfig in moments).

We'll touch base on mochi/other-tests timing/plan when we get closer to enabling those anywhere.

Unknown yet on what reftests will fail here, but we can always hide the suites and/or disable tests.

While I'm at it, I noticed we did not explicitly avoid esr10 as well, but I think that was already covered by not building there, but better-safe-than-sorry.
Comment 2 Aki Sasaki [:aki] 2012-09-13 15:44:54 PDT
Comment on attachment 661024 [details] [diff] [review]
[configs] do-it

esr10 has 'lock_platforms' set, but its addition is harmless.
Comment 3 Justin Wood (:Callek) 2012-09-13 15:48:50 PDT
Comment on attachment 661024 [details] [diff] [review]
[configs] do-it

http://hg.mozilla.org/build/buildbot-configs/rev/6581ccd98aff
Comment 4 Alex Keybl [:akeybl] 2012-09-13 16:01:26 PDT
Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the timeline there?
Comment 5 Aki Sasaki [:aki] 2012-09-13 16:52:43 PDT
armv6 reftests should be live on aurora+beta.

(In reply to Alex Keybl [:akeybl] from comment #4)
> Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the
> timeline there?

I think this bug will suffice if we keep it open.
Comment 6 Armen Zambrano [:armenzg] - Engineering productivity 2012-09-14 06:44:15 PDT
(In reply to Aki Sasaki [:aki] from comment #5)
> armv6 reftests should be live on aurora+beta.
> 
> (In reply to Alex Keybl [:akeybl] from comment #4)
> > Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the
> > timeline there?
> 
> I think this bug will suffice if we keep it open.

Joel or Brad might be able to answer where it is being worked on and depend on them.
Comment 7 Joel Maher (:jmaher) 2012-09-14 06:45:52 PDT
this bug was filed when we were running XUL android and didn't have the bandwidth to manage so many.  The only thing holding us back from more tests is planning for capacity.
Comment 8 Armen Zambrano [:armenzg] - Engineering productivity 2012-09-14 07:00:34 PDT
I see the reftests running:
https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=armv6
https://tbpl.mozilla.org/?tree=Mozilla-Beta&jobname=armv6

It looks very pretty :)
Comment 9 Alex Keybl [:akeybl] 2012-09-14 09:09:15 PDT
(In reply to Joel Maher (:jmaher) from comment #7)
> this bug was filed when we were running XUL android and didn't have the
> bandwidth to manage so many.  The only thing holding us back from more tests
> is planning for capacity.

Can we agree upon some cadence (daily, every x check-ins) for nightly/aurora and do this per-checkin for beta? We don't need to all the tests done all the time, and I'd like to try to get unblocked in time for Beta 16.
Comment 10 Aki Sasaki [:aki] 2012-09-14 11:27:51 PDT
(In reply to Alex Keybl [:akeybl] from comment #0)
> Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we
> enable all tests on Aurora/Beta, at some reasonable cadence? Thanks!

Alex:
for "all tests", do you mean all unit tests, or unit tests + talos (perf numbers) ?
Comment 11 Joel Maher (:jmaher) 2012-09-14 11:34:52 PDT
I want to make sure if we are running talos, that the results get published as a different os/platform/version combination so we don't have armv6 and armv7 builds running the same tests and reporting different numbers.  We worked around this when we switched from xul to native by adding a flag to talos so it would report a different machine name (which is a 1:1 mapping of the platform/os/version).
Comment 12 Alex Keybl [:akeybl] 2012-09-14 12:18:21 PDT
(In reply to Aki Sasaki [:aki] from comment #10)
> (In reply to Alex Keybl [:akeybl] from comment #0)
> > Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we
> > enable all tests on Aurora/Beta, at some reasonable cadence? Thanks!
> 
> Alex:
> for "all tests", do you mean all unit tests, or unit tests + talos (perf
> numbers) ?

I care most about unit tests for release, as user feedback on performance will be more important than talos numbers.
Comment 13 Aki Sasaki [:aki] 2012-09-14 12:30:48 PDT
Created attachment 661318 [details] [diff] [review]
run all unit tests on armv6

This will make android-armv6 run the exact same unit test suites as android (armv7).
Comment 14 Justin Wood (:Callek) 2012-09-14 12:42:43 PDT
Comment on attachment 661318 [details] [diff] [review]
run all unit tests on armv6

For those following along at home, this will make these all run on all branches while we're at it, and turn on mochi-1..8 and crashtest, jsreftest, and xpc
Comment 15 Aki Sasaki [:aki] 2012-09-14 12:44:25 PDT
Comment on attachment 661318 [details] [diff] [review]
run all unit tests on armv6

http://hg.mozilla.org/build/buildbot-configs/rev/092fc3e36d78

This should go live on Monday's reconfig.
Comment 16 Andrew McCreight [:mccr8] 2012-09-18 11:34:47 PDT
It looks like there may be a permaorange here:
26108 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug560780.html | got the mousedown events - got 1, expected 12

My push was the first that M1 was run on, so it is also possible that I somehow broke this test, though that seems unlikely. I can try backing it out if that will help.
Comment 17 Phil Ringnalda (:philor) 2012-09-18 11:39:58 PDT
You didn't. Hidden on inbound, aurora, beta and try.
Comment 18 Phil Ringnalda (:philor) 2012-09-18 20:19:10 PDT
Filed bug 792300, and hid them on every other trunk tree (except twigs, which always get the short end of the stick).
Comment 19 Alex Keybl [:akeybl] 2012-10-03 14:23:32 PDT
My understanding is that ARMv6 tests are running on ARMv7 (using the correct ARMv6 JS configuration). Resolved/fixed?
Comment 20 Justin Wood (:Callek) 2012-10-03 15:16:17 PDT
(In reply to Alex Keybl [:akeybl] from comment #19)
> My understanding is that ARMv6 tests are running on ARMv7 (using the correct
> ARMv6 JS configuration). Resolved/fixed?

They are, but a handful of suites are still hidden in production due to test and code issues.

Do you want this left open to track that, or would you rather a new bug to handle those "unhide" issues.
Comment 21 Alex Keybl [:akeybl] 2012-10-04 09:58:05 PDT
(In reply to Justin Wood (:Callek) from comment #20)
> (In reply to Alex Keybl [:akeybl] from comment #19)
> > My understanding is that ARMv6 tests are running on ARMv7 (using the correct
> > ARMv6 JS configuration). Resolved/fixed?
> 
> They are, but a handful of suites are still hidden in production due to test
> and code issues.
> 
> Do you want this left open to track that, or would you rather a new bug to
> handle those "unhide" issues.

I think we can resolve this bug, we'll track the three blockers for FF17.
Comment 22 Mats Palmgren (:mats) 2012-10-14 11:48:30 PDT
I suspect that all of these armv6 test failures are due to the tool chain bug
as explained in bug 790624 comment 17.  You can confirm this by doing a test
run with a custom mozconfig that has --enable-optimize that excludes -Os.
Comment 23 Mats Palmgren (:mats) 2012-10-28 16:23:20 PDT
Fwiw, there's still a lot of random crashes in the armv6 test runs, and some
on other Android too.  It appears we get no crash stacks in the log file.
Are these crashes being tracked somewhere?
Comment 24 Justin Wood (:Callek) 2012-10-28 16:40:55 PDT
(In reply to Mats Palmgren [:mats] from comment #23)
>  It appears we get no crash stacks in the log file.

No stacks in log could be one of two issues:

The first is Bug 795206 which is a releng bug, due to the controlling system the tegra jobs run from, and is on my plate to fix.

The second could be a host OS (android) crash which we don't yet have a good (any?) way to automate getting a stack at this point.
Comment 25 Phil Ringnalda (:philor) 2012-10-29 01:15:58 PDT
But what you are most likely to mean is ye olde http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=732325&entireHistory=true&tree=trunk, which since the October explosion has mostly been eating a shutdown crash. I wouldn't be particularly surprised if it's the same shutdown crash that the reftest harness has had all along, which used to be eaten by just showing as a green run unless someone happened to look at the log.
Comment 26 Justin Wood (:Callek) 2012-11-03 18:51:39 PDT
I'm going to resolve this, as a fixed thing -- we have Armv6 tests on every branch we have Armv6 builds at the moment, we might want to eventually expand the amount of tests we run, and we do intend to switch to armv6 devices in the future, but I'm calling *this* bug done.

Note You need to log in before you can comment on or make changes to this bug.