655046 - Android tests not being run on try; logs and builds in separate dirs

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris Cooper [:coop] (he/him)]

jfkthame reported in IRC this morning that he had not received any unittests for his Android build on Try despite specifying the following syntax:

 try: -p android-r7 -u all

http://hg.mozilla.org/try/rev/290b0456b5f1

I was unable to trigger follow-up tests for him with try_sendchange.py. After doing some quick digging, I noticed that Android builds are ending up in sub dirs that look like this:

/pub/mozilla.org/firefox/try-builds/[e-mail]-290b0456b5f1/try-mob-andrd-r7-bld/

...but the build logs end up here:

/pub/mozilla.org/firefox/try-builds/[e-mail]-290b0456b5f1/try-android-r7/

AFAICT, the tools are looking for builds+tests in the try-android-r7/ dir and not finding them.

However, it may not be so simple. cpearce ended up with Android tests for another non-Android-specific push shortly before this:

http://tbpl.mozilla.org/?tree=Try&rev=b96a33ce33c2

Maybe constraining the platform causes the tools look for builds in the wrong dir?

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

A little digging and it looks like http://hg.mozilla.org/build/buildbotcustom/rev/119176006f3e broke this, will look into a fix.

Comment 4

[Aki Sasaki (not active)]

Attachment: set user to %(who)s on mobile try sendchanges

So "Android tests not being run on try" is incorrect.
This is part of bug 653736, which causes buildbot to email "sendchange" about the mobile try builds rather than the submitter.

This patch should fix that portion; I still need to investigate upload directories and trychooser.

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 532302 [details] [diff] [review]
set user to %(who)s on mobile try sendchanges

There are more buildbotcustom changes for mobile try uploads, and possible try syntax. Removing r? flag for now.

Comment 6

[Aki Sasaki (not active)]

Attachment: try sendchange fix, for now

This adds android-r7 to try_sendchange.py, and adds .eabi-arm.apk as a valid build extension (not just .apk because that has 2 matches).

This is dependent on uploads going to try-android-r7/ instead of try-mob-andrd-r7-bld (patch incoming).

This will need to s,android-r7,android, when John Ford lands his MercurialBuildFactory patch.

To test, I softlinked the contents of try-mob-andrd-r7-bld/. into try-android-r7/. in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-d8078fc9279e/ and tested using --dry-run :

(bb08)deathduck:~/src/android-try/build-tools/buildfarm/maintenance [14:04:22]
1299$ python try_sendchange.py [e-mail]-d8078fc9279e --dry-run -p android-r7 -t all
Scanning ftp...

Found test pkg for android-r7
Found build for android-r7

What will be sent:

buildbot sendchange --master production-master01.build.mozilla.org:9009 --branch try-android-r7-talos --revision d8078fc9279e --comment "try: --t all" --user [e-mail] http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-d8078fc9279e/try-android-r7/fennec-6.0a1.en-US.eabi-arm.apk

Comment 7

[Aki Sasaki (not active)]

Attachment: fix mobile try upload directory and set user to %(who)s

In http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-d8078fc9279e/try-android-r7/try-mob-android-r7-build-build256.txt.gz (chosen at random), I noticed that during make upload,

POST_UPLOAD_CMD=post_upload.py --tinderbox-builds-dir try-android-r7 -p mobile -i 20110512124038 --revision d8078fc9279e --who [e-mail] --builddir try-mob-andrd-r7-bld --release-to-try-builds

So the --tinderbox-builds-dir is exactly what we want, and the --builddir is wrong.

Setting |uploadArgs['builddir'] = tinderboxBuildsDir| inside |if self.enable_try:| is pretty easy after that.

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 532348 [details] [diff] [review]
fix mobile try upload directory and set user to %(who)s

right on!

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 532348 [details] [diff] [review]
fix mobile try upload directory and set user to %(who)s

http://hg.mozilla.org/build/buildbotcustom/rev/6812cce361f7

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 532346 [details] [diff] [review]
try sendchange fix, for now

http://hg.mozilla.org/build/tools/rev/a678252151f1

Comment 13

[Armen [:armenzg]]

Also see bug 657357 for a similar issue if not the same (the developer indicated "reftest" rather than "reftest-1" and "reftest-2").

I will try to see if I can do a reconfig and pick aki's change up.

Comment 14

[Armen [:armenzg]]

Reconfigured masters this morning with this change.

Comment 15

[Aki Sasaki (not active)]

pm02-try was missing the reconfig -- did so today, and pushed another android/maemo build with all tests to try to test.

Any builds that started before this reconfig may still have the wrong upload/sendchange behavior.

Comment 16

[Jeff Muizelaar [:jrmuizel]]

Not sure if this is supposed to be fixed yet, but it still seems broken: http://tbpl.mozilla.org/?tree=Try&rev=396b9ecc8e04

Comment 17

[Aki Sasaki (not active)]

Yeah, I noticed that yesterday while trying to verify my fix.
The upload looks fixed; the sendchanges look right; the tests seem like they're not triggering.

I need to trace this.

Comment 19

[Aki Sasaki (not active)]

Try is building and running android tests when the comment is try: -a. When specifying android separately, it doesn't work.

I'm going to guess this is at least partially due to bug 592078.

Comment 20

[Aki Sasaki (not active)]

Ok, as I guessed, the discrepancy of naming the platform 'android-r7' in the mozilla/config.py dictionary and 'android' in the mozilla-tests/config.py dictionary caused problems in trychooser.

I tested a patch in staging where I renamed mozilla-tests/config.py's 'android' to 'android-r7' and that triggered tests.

Pretty soon, bug 557260 is going to rename the dictionary key in mozilla/config.py to 'linux-android' while making user-facing strings say 'android'.

I can push a temporary fix in the meantime that changes the 'android' strings in mozilla-tests/config.py to 'android-r7' and then change them to 'linux-android' when bug 557260 lands, or I can wait until bug 557260 lands and just change it once.

Are people needing this functionality sooner, or ok to wait til the end of the week?

Comment 21

[Aki Sasaki (not active)]

Attachment: mozilla-tests android -> android-r7

Comment 22

[Aki Sasaki (not active)]

Attachment: trychooser: accept android-r7, linux-android, or android as valid platforms

Comment 23

[Aki Sasaki (not active)]

Attachment: mozilla-tests android -> linux-android

Comment 24

[Aki Sasaki (not active)]

Attachment: trychooser: accept android-r7, linux-android, or android as valid platforms (linux-android version)

Comment 25

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 534638 [details] [diff] [review]
trychooser: accept android-r7, linux-android, or android as valid platforms

ugly, but if it works...it will have to do until there's one config file to rule them all :)

Comment 26

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 534643 [details] [diff] [review]
trychooser: accept android-r7, linux-android, or android as valid platforms (linux-android version)

Review of attachment 534643 [details] [diff] [review]:
-----------------------------------------------------------------

looks good to me!

Comment 27

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 534642 [details] [diff] [review]
mozilla-tests android -> linux-android

Review of attachment 534642 [details] [diff] [review]:
-----------------------------------------------------------------

as implemented, i don't think mozilla-tests knows about stage_platform or stage_product (correct me if i am wrong), so what is going to happen is that logs are going to upload to linux-android in /firefox/ instead of android in /mobile/ (i think). This is not a regression, just a change in behaviour.
In order to get the logs to upload to android, we could set a stage_platform and stage_product factory variable.  Those variables could be used to set builder properties and in turn used in the log upload command generation function, similar to the patch in 557260.

Marking as r+ because I don't think this is regressing behaviour and will normalize mozilla/ and mozilla-tests/ platform naming!

Thanks for writing the patch.

Comment 28

[Aki Sasaki (not active)]

Android tests should now be triggering properly.
I still need to deal with test log upload locations; I'll finish that off in bug 637838.