865677 - rerun android panda tests on newly rewired chassis with fennec 19.0

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

The results from running android tests on the newly rewired chassis (bug 860028) don't show the improvement that we were expecting.  

https://wiki.mozilla.org/Mobile/Testing/04_24_13#A_Team

Last week, the failure rates on the tegras also increased so ateam suspects that there may be an issue with the product itself since we ran the smoketests.

Ateam has requested that I run some tests with this test zip

http://stage.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-release-android/1364033724/fennec-19.0.2.en-US.android-arm.tests.zip

and the 19.0 apk in the smoketest

http://people.mozilla.org/~jmaher/smoketest/smoketest_binaries.zip

on my dev master and see what the results look like.  In addition, we'll avoid running talos (rp, rpr, rck, rck2) and robocop 1,2 because they are flaky.

Comment 1

[Kim Moir [:kmoir] ET]

Builds are running here 
http://dev-master01.build.scl1.mozilla.com:8036/one_line_per_build

I ended up having to use the 19.0release apk and associated test.zip.  jmaher used a different apk for his smoketesting but assures me that this is close enough.  Otherwise the tests.zip and apk don't match and we see orange results.

Comment 2

[Kim Moir [:kmoir] ET]

The builds over the weekend had too many failed pandas.  I removed the bad pandas from the configs (295,300,301,302,305,306,334,337,338,340,344) and restarted them.  Builds that finished from Apr 29 10:42 onward are looking better, no purple at least.

http://dev-master01.build.scl1.mozilla.com:8036/one_line_per_build?numbuilds=200

Comment 3

[Kim Moir [:kmoir] ET]

I looked at the test results for the 19.0 run on my dev master.

670 tests run since yesterday
27 purple - these were limited to 2 pandas 342 and 333
157 orange - Mochitest 1-3 and jsreftest 1-3. 48 of these were mochitest 1 which I believe are hidden now due to bug 865443.  Is this enough data for the test? Seems like there's still a pretty high failure rate.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

21.5% - including everything
18.9% - without the 2 pandas
14.0% - ignoring 2 pandas and m1

on tbpl we have a lot of blue and red results as well.  The failure rate seems higher than normal, at least for oranges.  One thought is the smoketest should be updated to include more realistic scenarios.  The original intention of the smoketest was to prove the hardware platform and the tools using it are stable.  Knowing that we are seeing oranges instead of red/purple/blue, I would say the hardware is stable...our goal was to have no [random] oranges while we test.

I am still not sure what to make of this.

Comment 5

[Kim Moir [:kmoir] ET]

I'm going to close this, please reopen if you require additional testing :-)