836162 - Kill test sendchanges from android* release builds

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Phil Ringnalda (:philor)]

Load https://secure.pub.build.mozilla.org/buildapi/running and sort by Submitted at. Just below the 5 zombies, you'll see a

release-mozilla-beta 44ad9c166d46 Android Tegra 250 release-mozilla-beta opt test jsreftest-1

which was submitted at 2013-01-23 09:35:05, with a Running since of somewhere within 10 minutes or so of when you loaded the page. Come back in 10 minutes or so, and you'll see it with a new Running since, because we've been infinitely retrying that job for a week now.

Comment 1

[Chris AtLee [:catlee]]

killed

Comment 2

[[e-mail] (:bhearsum)]

This is something happens with tegras sometimes AFAIK.

Comment 3

[Phil Ringnalda (:philor)]

Well, not really. If we set up a p0 infinite retry every single time we release a beta, then before long we'll be out of tegras. Apparently last week we only got one, but now from today we got five more.

Comment 4

[Phil Ringnalda (:philor)]

May well be two bugs if it's the same retry we've been suffering through everywhere that didn't get the Android signing patch, though, one for why release jobs are hitting it, and a separate non-Automation (Release) bug about turning the error message into something that doesn't retry, since right now just pushing to try from an old repo sets up every Android job infinitely retrying.

Comment 5

[Phil Ringnalda (:philor)]

Now we're down to just three (I presume thanks to tegras that disconnect before they get to the part that retries). One of the three is http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/None-candidates/buildNone/logs/release-mozilla-beta_tegra_android-armv6_test-jsreftest-1-bm19-tests1-tegra-build116.txt.gz, which is indeed the same symptom as the pre-signing-patch failures, "Remote Device Error: updateApp() call failed - exiting".

So we could treat the symptom by just making that failure not be a "Remote Device Error:" so that we don't RETRY from it, but I'm sure someone will want to look into why we're hitting it.

Comment 6

[Aki Sasaki (not active)]

I thought about this before, and we could test a fennec apk with any signature if we re-signed it with a testing key before testing.  (All tegras+pandas would need to have SUT agents signed with the same key, so unless the testing key == the nightly key this would be a very large reimaging party.)

We probably need to turn off sendchanges from release builds for the time being.  Afaik we never really looked at the results before releasing anyway.

Comment 7

[Aki Sasaki (not active)]

Ok, I was wrong about release signing preventing installation (Callek manually verified on a tegra).

Happy to be wrong.  Since this isn't true, bug 836861 should hopefully get us working.

Comment 8

[Aki Sasaki (not active)]

Turns out most tegras can't install release-signed builds.
My current theory:

Anything that has a nightly-signed apk installed will be unable to install a release-signed apk.  As tegras test more and more things (esp on try) the subset of tegras that can install release-signed apks shrink to near-zero.

Callek could have tested on a tegra that was able to install a release-signed apk in comment 7.

Given this, and given that we don't actually look at the release build tests, let's kill the release build sendchanges.

Comment 9

[Aki Sasaki (not active)]

This may be as simple as setting this line in our various fennec release configs to False:
http://hg.mozilla.org/build/buildbot-configs/file/5a2512247972/mozilla/release-fennec-mozilla-beta.py#l75

Doing some investigation.

Comment 10

[Aki Sasaki (not active)]

Attachment: disable android release unittests

Running a dump_masters.sh to compare.  Pretty sure this is it, though.

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

This also disables make package-tests.
Not sure if this is an issue.

Comment 12

[Justin Wood (:Callek)]

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

Review of attachment 710916 [details] [diff] [review]:
-----------------------------------------------------------------

sheep it

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

http://hg.mozilla.org/build/buildbot-configs/rev/1da41328b1c9

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 710916 [details] [diff] [review]
disable android release unittests

Transplanted: http://hg.mozilla.org/build/buildbot-configs/rev/b917827a70af

Comment 15

[Aki Sasaki (not active)]

This should be fixed as of the releases spinning today&tomorrow.