791103 - ARMv6 tests on ARMv7 for Beta 16

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Alex Keybl [:akeybl]]

Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we enable all tests on Aurora/Beta, at some reasonable cadence? Thanks!

Comment 1

[Justin Wood (:Callek)]

Attachment: [configs] do-it

This adds reftests to aurora/beta as of today (aki should be doing a reconfig in moments).

We'll touch base on mochi/other-tests timing/plan when we get closer to enabling those anywhere.

Unknown yet on what reftests will fail here, but we can always hide the suites and/or disable tests.

While I'm at it, I noticed we did not explicitly avoid esr10 as well, but I think that was already covered by not building there, but better-safe-than-sorry.

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 661024 [details] [diff] [review]
[configs] do-it

esr10 has 'lock_platforms' set, but its addition is harmless.

Comment 3

[Justin Wood (:Callek)]

Comment on attachment 661024 [details] [diff] [review]
[configs] do-it

http://hg.mozilla.org/build/buildbot-configs/rev/6581ccd98aff

Comment 4

[Alex Keybl [:akeybl]]

Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the timeline there?

Comment 5

[Aki Sasaki (not active)]

armv6 reftests should be live on aurora+beta.

(In reply to Alex Keybl [:akeybl] from comment #4)
> Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the
> timeline there?

I think this bug will suffice if we keep it open.

Comment 6

[Armen [:armenzg]]

(In reply to Aki Sasaki [:aki] from comment #5)
> armv6 reftests should be live on aurora+beta.
> 
> (In reply to Alex Keybl [:akeybl] from comment #4)
> > Where can we track enabling mochi/other-tests on ARMv7 for ARMv6 and the
> > timeline there?
> 
> I think this bug will suffice if we keep it open.

Joel or Brad might be able to answer where it is being worked on and depend on them.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

this bug was filed when we were running XUL android and didn't have the bandwidth to manage so many.  The only thing holding us back from more tests is planning for capacity.

Comment 8

[Armen [:armenzg]]

I see the reftests running:
https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=armv6
https://tbpl.mozilla.org/?tree=Mozilla-Beta&jobname=armv6

It looks very pretty :)

Comment 9

[Alex Keybl [:akeybl]]

(In reply to Joel Maher (:jmaher) from comment #7)
> this bug was filed when we were running XUL android and didn't have the
> bandwidth to manage so many.  The only thing holding us back from more tests
> is planning for capacity.

Can we agree upon some cadence (daily, every x check-ins) for nightly/aurora and do this per-checkin for beta? We don't need to all the tests done all the time, and I'd like to try to get unblocked in time for Beta 16.

Comment 10

[Aki Sasaki (not active)]

(In reply to Alex Keybl [:akeybl] from comment #0)
> Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we
> enable all tests on Aurora/Beta, at some reasonable cadence? Thanks!

Alex:
for "all tests", do you mean all unit tests, or unit tests + talos (perf numbers) ?

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

I want to make sure if we are running talos, that the results get published as a different os/platform/version combination so we don't have armv6 and armv7 builds running the same tests and reporting different numbers.  We worked around this when we switched from xul to native by adding a flag to talos so it would report a different machine name (which is a 1:1 mapping of the platform/os/version).

Comment 12

[Alex Keybl [:akeybl]]

(In reply to Aki Sasaki [:aki] from comment #10)
> (In reply to Alex Keybl [:akeybl] from comment #0)
> > Aki let me know that reftests have been enabled for ARMv6 on nightly. Can we
> > enable all tests on Aurora/Beta, at some reasonable cadence? Thanks!
> 
> Alex:
> for "all tests", do you mean all unit tests, or unit tests + talos (perf
> numbers) ?

I care most about unit tests for release, as user feedback on performance will be more important than talos numbers.

Comment 13

[Aki Sasaki (not active)]

Attachment: run all unit tests on armv6

This will make android-armv6 run the exact same unit test suites as android (armv7).

Comment 14

[Justin Wood (:Callek)]

Comment on attachment 661318 [details] [diff] [review]
run all unit tests on armv6

For those following along at home, this will make these all run on all branches while we're at it, and turn on mochi-1..8 and crashtest, jsreftest, and xpc

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 661318 [details] [diff] [review]
run all unit tests on armv6

http://hg.mozilla.org/build/buildbot-configs/rev/092fc3e36d78

This should go live on Monday's reconfig.

Comment 16

[Andrew McCreight [:mccr8]]

It looks like there may be a permaorange here:
26108 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_bug560780.html | got the mousedown events - got 1, expected 12

My push was the first that M1 was run on, so it is also possible that I somehow broke this test, though that seems unlikely. I can try backing it out if that will help.

Comment 17

[Phil Ringnalda (:philor)]

You didn't. Hidden on inbound, aurora, beta and try.

Comment 18

[Phil Ringnalda (:philor)]

Filed bug 792300, and hid them on every other trunk tree (except twigs, which always get the short end of the stick).

Comment 19

[Alex Keybl [:akeybl]]

My understanding is that ARMv6 tests are running on ARMv7 (using the correct ARMv6 JS configuration). Resolved/fixed?

Comment 20

[Justin Wood (:Callek)]

(In reply to Alex Keybl [:akeybl] from comment #19)
> My understanding is that ARMv6 tests are running on ARMv7 (using the correct
> ARMv6 JS configuration). Resolved/fixed?

They are, but a handful of suites are still hidden in production due to test and code issues.

Do you want this left open to track that, or would you rather a new bug to handle those "unhide" issues.

Comment 21

[Alex Keybl [:akeybl]]

(In reply to Justin Wood (:Callek) from comment #20)
> (In reply to Alex Keybl [:akeybl] from comment #19)
> > My understanding is that ARMv6 tests are running on ARMv7 (using the correct
> > ARMv6 JS configuration). Resolved/fixed?
> 
> They are, but a handful of suites are still hidden in production due to test
> and code issues.
> 
> Do you want this left open to track that, or would you rather a new bug to
> handle those "unhide" issues.

I think we can resolve this bug, we'll track the three blockers for FF17.

Comment 22

[Mats Palmgren (inactive)]

I suspect that all of these armv6 test failures are due to the tool chain bug
as explained in bug 790624 comment 17.  You can confirm this by doing a test
run with a custom mozconfig that has --enable-optimize that excludes -Os.

Comment 23

[Mats Palmgren (inactive)]

Fwiw, there's still a lot of random crashes in the armv6 test runs, and some
on other Android too.  It appears we get no crash stacks in the log file.
Are these crashes being tracked somewhere?

Comment 24

[Justin Wood (:Callek)]

(In reply to Mats Palmgren [:mats] from comment #23)
>  It appears we get no crash stacks in the log file.

No stacks in log could be one of two issues:

The first is Bug 795206 which is a releng bug, due to the controlling system the tegra jobs run from, and is on my plate to fix.

The second could be a host OS (android) crash which we don't yet have a good (any?) way to automate getting a stack at this point.

Comment 25

[Phil Ringnalda (:philor)]

But what you are most likely to mean is ye olde http://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=732325&entireHistory=true&tree=trunk, which since the October explosion has mostly been eating a shutdown crash. I wouldn't be particularly surprised if it's the same shutdown crash that the reftest harness has had all along, which used to be eaten by just showing as a green run unless someone happened to look at the log.

Comment 26

[Justin Wood (:Callek)]

I'm going to resolve this, as a fixed thing -- we have Armv6 tests on every branch we have Armv6 builds at the moment, we might want to eventually expand the amount of tests we run, and we do intend to switch to armv6 devices in the future, but I'm calling *this* bug done.