789652 - schedule B2G builds to be tested within emulator-with-codecs

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting w/cjones, jgriffin:

Once emulator toolchain is deployed, bug#789651, then we should be able to schedule tests run on each of the Armv7a ICS Opt/Debug builds. This bug is to track enabling those tests run per build.

cjones/jgriffin: For each of Opt, Debug builds, can you give us a list of which the test suites are expected to run green, so we know which testsuites to schedule and display on tbpl.m.o?

Comment 1

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

The test results from the runs on jenkins infra are reported here

http://brasstacks.mozilla.com/autolog/?tree=b2g&source=autolog

but we only do opt builds there.  We have some bugs to shake out.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per followup call w/jgriffin: 

(In reply to John O'Duinn [:joduinn] from comment #0)
> per meeting w/cjones, jgriffin:
> 
> Once emulator toolchain is deployed, bug#789651, then we should be able to
> schedule tests run on each of the Armv7a ICS Opt/Debug builds. This bug is
> to track enabling those tests run per build.
Note: the existing "Armv7a ICS opt" and "Armv7a ICS debug" builds are good here. No need to generate additional builds, now we'll just be testing those existing builds!



> cjones/jgriffin: For each of Opt, Debug builds, can you give us a list of
> which the test suites are expected to run green, so we know which testsuites
> to schedule and display on tbpl.m.o?
Exact list of testsuites is still being worked out as daily bustages are fixed; jgriffin will give us a proper list when everything else is ready.

Note: jgriffin is working in bug#789976 to enable B2G builds to be installed (using ADB?) as part of each test job. Once bug#789976 is completed, we can use this bug (bug#789652) to track the RelEng scheduling and any related mozharness work to get this running in production.

Comment 4

[Chris AtLee [:catlee]]

Is there any restriction on the type of machine these need to run on?

Comment 5

[Jonathan Griffin (:jgriffin)]

In order to get the most out of reftests (and maybe some mochitests), they should be run on machines with real GPU's (not VM's).  Linux32 or 64 is the most tested and stable platform for these.

Comment 6

[Rail Aliiev [:rail]]

Failing:

/tools/buildbot/bin/python scripts/scripts/marionette.py --cfg marionette/prod_config.py
 in dir /builds/slave/talos-slave/test/. (timeout 1200 secs) (maxTime 3600 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', 'scripts/scripts/marionette.py', '--cfg', 'marionette/prod_config.py']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-OI3H6G/Render
  Apple_Ubiquity_Message=/tmp/launch-yLXv9I/Apple_Ubiquity_Message
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PROPERTIES_FILE=/builds/slave/talos-slave/test/buildprops.json
  PWD=/builds/slave/talos-slave/test
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-VkvkOj/Listeners
  TMPDIR=/var/folders/xx/59ykv66n1hj3nt4n04tr9sh400000w/T/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1C:0:0
 using PTY: False
Traceback (most recent call last):
  File "scripts/scripts/marionette.py", line 244, in <module>
    marionetteTest = MarionetteTest()
  File "scripts/scripts/marionette.py", line 115, in __init__
    'require_test_zip': True,})
  File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 634, in __init__
    self.new_log_obj(default_log_level=default_log_level)
  File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 761, in new_log_obj
    dirs = self.query_abs_dirs()
  File "scripts/scripts/marionette.py", line 145, in query_abs_dirs
    dirs['abs_work_dir'], 'gecko')
KeyError: 'abs_work_dir'
program finished with exit code 1
elapsedTime=0.292439

Comment 7

[Armen [:armenzg]]

IIUC correctly I have uploaded the emulator to:
relengweb1.dmz.scl3.mozilla.com:/var/www/html/runtime-binaries/tooltool/sha512
The filename should is the sha512sum of the file.
[[e-mail]3 sha512]# sha512sum emulator-arm_linux_2012-10-05.zip 
69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf  emulator-arm_linux_2012-10-05.zip
[[e-mail]3 sha512]# mv emulator-arm_linux_2012-10-05.zip 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf

The mozharness script would need a manifest like this:
b2g/config/tooltool-manifests/ics.manifest content
=======================================================
[
{
"size": 542093126, 
"digest": "69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf", 
"algorithm": "sha512", 
"filename": "TBD"
},

Comment 8

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano G. [:armenzg] from comment #7)
> IIUC correctly I have uploaded the emulator to:
> relengweb1.dmz.scl3.mozilla.com:/var/www/html/runtime-binaries/tooltool/
> sha512
> The filename should is the sha512sum of the file.
> [[e-mail]3 sha512]# sha512sum
> emulator-arm_linux_2012-10-05.zip 
> 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff6433369
> 2a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf 
> emulator-arm_linux_2012-10-05.zip
> [[e-mail]3 sha512]# mv emulator-arm_linux_2012-10-05.zip
> 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff6433369
> 2a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf
> 
> The mozharness script would need a manifest like this:
> b2g/config/tooltool-manifests/ics.manifest content
> =======================================================
> [
> {
> "size": 542093126, 
> "digest":
> "69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff643336
> 92a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf", 
> "algorithm": "sha512", 
> "filename": "TBD"
> },

I'm not sure how the mozharness script can use this to download the emulator.  Is there a url it's available at?

Comment 9

[Armen [:armenzg]]

We can either add tooltool.py usage to the mozharness (which I would rather leave out of scope and follow up later) or grab it directly from here:
http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf

I know, it's an ugly URL but would that work for now?

tooltool support [1] is the same thing as talos.json [2] but more sophisticated.

I have not done any of this myself but I can help figure things out.

http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/tooltool.py
http://hg.mozilla.org/mozilla-central/file/default/testing/talos/talos.json

Comment 10

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano G. [:armenzg] from comment #9)
> We can either add tooltool.py usage to the mozharness (which I would rather
> leave out of scope and follow up later) or grab it directly from here:
> http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/
> 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff6433369
> 2a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf
> 
> I know, it's an ugly URL but would that work for now?
> 
Yep, that works.  Thanks.

Comment 11

[Aki Sasaki (not active)]

Attachment: schedule b2g mochitests, rename them

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 673447 [details] [diff] [review]
schedule b2g mochitests, rename them

Review of attachment 673447 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +27,5 @@
>  builder_prefix = "b2g "
>  
>  PLATFORMS['ics_armv7a_gecko']['slave_platforms'] = ['fedora-b2g']
>  PLATFORMS['ics_armv7a_gecko']['env_name'] = 'linux-perf'
> +PLATFORMS['ics_armv7a_gecko']['fedora-b2g'] = {'name': builder_prefix + "ics_armv7a_gecko emulator"}

Is the name changing here because we're testing inside of the emulator instead of in Fedora itself?

Comment 13

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #12)
> Is the name changing here because we're testing inside of the emulator
> instead of in Fedora itself?

I changed the name because the test was showing up in the Linux row on tbpl, and it makes more sense to show up in the B2G Arm opt row.

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #13)
> (In reply to Ben Hearsum [:bhearsum] from comment #12)
> > Is the name changing here because we're testing inside of the emulator
> > instead of in Fedora itself?
> 
> I changed the name because the test was showing up in the Linux row on tbpl,
> and it makes more sense to show up in the B2G Arm opt row.

OK. Are we sure we want them to be called ics_armv7a_gecko? If there's a better name we can make TBPL recognize it and sort them correctly. Maybe they should be "B2G Emulator" or something? I'm not fully plugged into this, so just throwing out suggestions.

Comment 15

[Aki Sasaki (not active)]

CCing sheriffs to see if they have a preference in naming.

Comment 16

[Aki Sasaki (not active)]

Comment on attachment 673447 [details] [diff] [review]
schedule b2g mochitests, rename them

Landing so the ateam has something to work with.
Happy to rename if/when we have something to rename to.

http://hg.mozilla.org/build/buildbot-configs/rev/30ab26725df6

Comment 17

[Aki Sasaki (not active)]

This is live.

Comment 18

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #14)
> OK. Are we sure we want them to be called ics_armv7a_gecko? If there's a
> better name we can make TBPL recognize it and sort them correctly. Maybe
> they should be "B2G Emulator" or something? I'm not fully plugged into this,
> so just throwing out suggestions.

Yeah, now they're showing up in the correct row but tbpl thinks they're builds.
So if we're going to need a tbpl patch anyway, I'd be happy to take the ics_armv7a_gecko out of the name.

Comment 19

[Ed Morley [:emorley]]

(In reply to Aki Sasaki [:aki] from comment #18)
> Yeah, now they're showing up in the correct row but tbpl thinks they're
> builds.
> So if we're going to need a tbpl patch anyway, I'd be happy to take the
> ics_armv7a_gecko out of the name.

My TBPL patches (awaiting review) in bug 786314 improve the situation slightly, however the main problem is that the naming of most things B2G is inconsistent

All of the other platforms currently use:
{OS_name} {optional_testing_platform_name_eg_Tegra} {tree_name} {build_or_test_name}

Whereas B2G use an assortment of different string orders and also space vs underscore vs hyphen naming:
b2g_cedar_ics_armv7a_gecko build
b2g_cedar_ics_armv7a_gecko-debug build
b2g_cedar_panda_dep
b2g ics_armv7a_gecko emulator cedar opt test marionette-webapi 

I'm happy to file the necessary bugs to get these straightened out (in both buildbot-configs & TBPL), but first:

1) Are the emulator builds supposed to be shown on the same line as "b2g_{treeName}_ics_armv7a_gecko build"? (As of today, now called "B2G Arm {opt,debug}")

2) Are the new "B2G Arm {opt,debug}" & "B2G Panda opt" rows on TBPL suitably named?

3) If we're wanting to get rid of "ics_armv7a_gecko" from the buildernames, what suggestions are their for its replacement (to differentiate them from panda builds?

Comment 20

[Aki Sasaki (not active)]

I might not be the best person to answer this, but aiui:

(In reply to Ed Morley (PTO/travelling until 24th Oct) [:edmorley UTC+1] from comment #19)
> 1) Are the emulator builds supposed to be shown on the same line as
> "b2g_{treeName}_ics_armv7a_gecko build"? (As of today, now called "B2G Arm
> {opt,debug}")

Yes.
I would guess we may have additional rows later (e.g. jb_armv7a_gecko for Jelly Bean might be a future row), so we may want to have a way to differentiate.
That isn't an issue currently.

> 2) Are the new "B2G Arm {opt,debug}" & "B2G Panda opt" rows on TBPL suitably
> named?

B2G Panda is fine.

B2G Arm seems fine for now; we may rename if/when we have multiple types of emulator builds.

> 3) If we're wanting to get rid of "ics_armv7a_gecko" from the buildernames,
> what suggestions are their for its replacement (to differentiate them from
> panda builds?

I think ics_armv7a_gecko will stay in the build names, and might go away from the test names.

(marionette-webapi and mochitests-1 through 3 are tests, but show up as builds atm.)

Comment 21

[Ed Morley [:emorley]]

Thank you for the quick reply :-)

(In reply to Aki Sasaki [:aki] from comment #20)
> I think ics_armv7a_gecko will stay in the build names, and might go away
> from the test names.

That would make them different from every other test buildername. We always list the OS/platform in the buildername for tests, since we need a way to associate them with a row on TBPL (and without ics_armv7a_gecko or similar, we'd be left with 'B2G', which could apply to multiple rows).

> (marionette-webapi and mochitests-1 through 3 are tests, but show up as
> builds atm.)

The reason they show up as builds isn't because of the ics_armv7a_gecko specifically, it's because the initial regex to match ics_armv7a_gecko wasn't specific enough, and will be fixed by the patches in bug 786314:
>-        /b2g.*_gecko/i.test(name) ? "B2G Gecko Build" :
>-        /b2g.*_panda_dep/i.test(name) ? "B2G Panda Image" :
>+        /b2g_.*_gecko(?:-debug)? build/i.test(name) ? "B2G Gecko Build" :
>+        /b2g_.*_panda_dep/i.test(name) ? "B2G Panda Image" :

One more question (sorry! :-)) - are we moving towards using underscores in buildernames across the board, or should they be removed from the B2G buildernames to be consistent with everything else?

Comment 22

[Aki Sasaki (not active)]

(In reply to Ed Morley (PTO/travelling until 24th Oct) [:edmorley UTC+1] from comment #21)
> Thank you for the quick reply :-)

Thank you.
> (In reply to Aki Sasaki [:aki] from comment #20)
> > I think ics_armv7a_gecko will stay in the build names, and might go away
> > from the test names.
> 
> That would make them different from every other test buildername. We always
> list the OS/platform in the buildername for tests, since we need a way to
> associate them with a row on TBPL (and without ics_armv7a_gecko or similar,
> we'd be left with 'B2G', which could apply to multiple rows).

Happy to keep them, then.

> > (marionette-webapi and mochitests-1 through 3 are tests, but show up as
> > builds atm.)
> 
> The reason they show up as builds isn't because of the ics_armv7a_gecko
> specifically, it's because the initial regex to match ics_armv7a_gecko
> wasn't specific enough, and will be fixed by the patches in bug 786314:
> >-        /b2g.*_gecko/i.test(name) ? "B2G Gecko Build" :
> >-        /b2g.*_panda_dep/i.test(name) ? "B2G Panda Image" :
> >+        /b2g_.*_gecko(?:-debug)? build/i.test(name) ? "B2G Gecko Build" :
> >+        /b2g_.*_panda_dep/i.test(name) ? "B2G Panda Image" :

Nice, thanks again.

> One more question (sorry! :-)) - are we moving towards using underscores in
> buildernames across the board, or should they be removed from the B2G
> buildernames to be consistent with everything else?

Hrm, probably the latter.  It may be as simple as tweaking the base_name's in http://hg.mozilla.org/build/buildbot-configs/file/a6af987ce749/mozilla/b2g_config.py to be more like the base_name's in http://hg.mozilla.org/build/buildbot-configs/file/a6af987ce749/mozilla/config.py .

Comment 23

[Ed Morley [:emorley]]

I've made a few tweaks to the TBPL patches in bug 786314 which will work around the buildername variations for now. These have just been pushed to production in bug 805201, so everything on Cedar shows up in TBPL properly now :-)

I'll file a separate bug for sorting on the buildbot side.

Comment 24

[Aki Sasaki (not active)]

These are scheduled and bug 805914 is landed.
Reftests and xpcshell are blocked due to external reasons; I can schedule them when they're ready.

I'm going to mark this resolved; please file new bugs for any new issues.