schedule B2G builds to be tested within emulator-with-codecs

RESOLVED FIXED

Status

Release Engineering
General Automation
P2
normal
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: joduinn, Assigned: aki)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [b2g][testing])

Attachments

(1 attachment)

per meeting w/cjones, jgriffin:

Once emulator toolchain is deployed, bug#789651, then we should be able to schedule tests run on each of the Armv7a ICS Opt/Debug builds. This bug is to track enabling those tests run per build.

cjones/jgriffin: For each of Opt, Debug builds, can you give us a list of which the test suites are expected to run green, so we know which testsuites to schedule and display on tbpl.m.o?
The test results from the runs on jenkins infra are reported here

http://brasstacks.mozilla.com/autolog/?tree=b2g&source=autolog

but we only do opt builds there.  We have some bugs to shake out.
Depends on: 789976
per followup call w/jgriffin: 

(In reply to John O'Duinn [:joduinn] from comment #0)
> per meeting w/cjones, jgriffin:
> 
> Once emulator toolchain is deployed, bug#789651, then we should be able to
> schedule tests run on each of the Armv7a ICS Opt/Debug builds. This bug is
> to track enabling those tests run per build.
Note: the existing "Armv7a ICS opt" and "Armv7a ICS debug" builds are good here. No need to generate additional builds, now we'll just be testing those existing builds!



> cjones/jgriffin: For each of Opt, Debug builds, can you give us a list of
> which the test suites are expected to run green, so we know which testsuites
> to schedule and display on tbpl.m.o?
Exact list of testsuites is still being worked out as daily bustages are fixed; jgriffin will give us a proper list when everything else is ready.

Note: jgriffin is working in bug#789976 to enable B2G builds to be installed (using ADB?) as part of each test job. Once bug#789976 is completed, we can use this bug (bug#789652) to track the RelEng scheduling and any related mozharness work to get this running in production.
Duplicate of this bug: 777534
Is there any restriction on the type of machine these need to run on?
Assignee: nobody → catlee
Priority: -- → P4
Whiteboard: [b2g][testing]
In order to get the most out of reftests (and maybe some mochitests), they should be run on machines with real GPU's (not VM's).  Linux32 or 64 is the most tested and stable platform for these.
Depends on: 790741
Depends on: 774535
Depends on: 792945
Depends on: 793240
Assignee: catlee → armenzg
Priority: P4 → P2
Failing:

/tools/buildbot/bin/python scripts/scripts/marionette.py --cfg marionette/prod_config.py
 in dir /builds/slave/talos-slave/test/. (timeout 1200 secs) (maxTime 3600 secs)
 watching logfiles {}
 argv: ['/tools/buildbot/bin/python', 'scripts/scripts/marionette.py', '--cfg', 'marionette/prod_config.py']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-OI3H6G/Render
  Apple_Ubiquity_Message=/tmp/launch-yLXv9I/Apple_Ubiquity_Message
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PROPERTIES_FILE=/builds/slave/talos-slave/test/buildprops.json
  PWD=/builds/slave/talos-slave/test
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-VkvkOj/Listeners
  TMPDIR=/var/folders/xx/59ykv66n1hj3nt4n04tr9sh400000w/T/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1C:0:0
 using PTY: False
Traceback (most recent call last):
  File "scripts/scripts/marionette.py", line 244, in <module>
    marionetteTest = MarionetteTest()
  File "scripts/scripts/marionette.py", line 115, in __init__
    'require_test_zip': True,})
  File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 634, in __init__
    self.new_log_obj(default_log_level=default_log_level)
  File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 761, in new_log_obj
    dirs = self.query_abs_dirs()
  File "scripts/scripts/marionette.py", line 145, in query_abs_dirs
    dirs['abs_work_dir'], 'gecko')
KeyError: 'abs_work_dir'
program finished with exit code 1
elapsedTime=0.292439
IIUC correctly I have uploaded the emulator to:
relengweb1.dmz.scl3.mozilla.com:/var/www/html/runtime-binaries/tooltool/sha512
The filename should is the sha512sum of the file.
[root@relengweb1.dmz.scl3 sha512]# sha512sum emulator-arm_linux_2012-10-05.zip 
69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf  emulator-arm_linux_2012-10-05.zip
[root@relengweb1.dmz.scl3 sha512]# mv emulator-arm_linux_2012-10-05.zip 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf

The mozharness script would need a manifest like this:
b2g/config/tooltool-manifests/ics.manifest content
=======================================================
[
{
"size": 542093126, 
"digest": "69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf", 
"algorithm": "sha512", 
"filename": "TBD"
},
(In reply to Armen Zambrano G. [:armenzg] from comment #7)
> IIUC correctly I have uploaded the emulator to:
> relengweb1.dmz.scl3.mozilla.com:/var/www/html/runtime-binaries/tooltool/
> sha512
> The filename should is the sha512sum of the file.
> [root@relengweb1.dmz.scl3 sha512]# sha512sum
> emulator-arm_linux_2012-10-05.zip 
> 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff6433369
> 2a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf 
> emulator-arm_linux_2012-10-05.zip
> [root@relengweb1.dmz.scl3 sha512]# mv emulator-arm_linux_2012-10-05.zip
> 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff6433369
> 2a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf
> 
> The mozharness script would need a manifest like this:
> b2g/config/tooltool-manifests/ics.manifest content
> =======================================================
> [
> {
> "size": 542093126, 
> "digest":
> "69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff643336
> 92a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf", 
> "algorithm": "sha512", 
> "filename": "TBD"
> },

I'm not sure how the mozharness script can use this to download the emulator.  Is there a url it's available at?
We can either add tooltool.py usage to the mozharness (which I would rather leave out of scope and follow up later) or grab it directly from here:
http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff64333692a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf

I know, it's an ugly URL but would that work for now?

tooltool support [1] is the same thing as talos.json [2] but more sophisticated.

I have not done any of this myself but I can help figure things out.

http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/tooltool.py
http://hg.mozilla.org/mozilla-central/file/default/testing/talos/talos.json
(In reply to Armen Zambrano G. [:armenzg] from comment #9)
> We can either add tooltool.py usage to the mozharness (which I would rather
> leave out of scope and follow up later) or grab it directly from here:
> http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512/
> 69cba761fc84f8db3b5f536c60027b6515acd5b3084156c67319bdb8e18b06170aedff6433369
> 2a80ea1ef8f9c5bdc6fc688b559703b59b5071aebb1bd6ffddf
> 
> I know, it's an ugly URL but would that work for now?
> 
Yep, that works.  Thanks.
Depends on: 800025
(Assignee)

Updated

6 years ago
Assignee: armenzg → aki
(Assignee)

Comment 11

6 years ago
Created attachment 673447 [details] [diff] [review]
schedule b2g mochitests, rename them
Attachment #673447 - Flags: review?(bhearsum)
Comment on attachment 673447 [details] [diff] [review]
schedule b2g mochitests, rename them

Review of attachment 673447 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/b2g_config.py
@@ +27,5 @@
>  builder_prefix = "b2g "
>  
>  PLATFORMS['ics_armv7a_gecko']['slave_platforms'] = ['fedora-b2g']
>  PLATFORMS['ics_armv7a_gecko']['env_name'] = 'linux-perf'
> +PLATFORMS['ics_armv7a_gecko']['fedora-b2g'] = {'name': builder_prefix + "ics_armv7a_gecko emulator"}

Is the name changing here because we're testing inside of the emulator instead of in Fedora itself?
Attachment #673447 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 13

6 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #12)
> Is the name changing here because we're testing inside of the emulator
> instead of in Fedora itself?

I changed the name because the test was showing up in the Linux row on tbpl, and it makes more sense to show up in the B2G Arm opt row.
(In reply to Aki Sasaki [:aki] from comment #13)
> (In reply to Ben Hearsum [:bhearsum] from comment #12)
> > Is the name changing here because we're testing inside of the emulator
> > instead of in Fedora itself?
> 
> I changed the name because the test was showing up in the Linux row on tbpl,
> and it makes more sense to show up in the B2G Arm opt row.

OK. Are we sure we want them to be called ics_armv7a_gecko? If there's a better name we can make TBPL recognize it and sort them correctly. Maybe they should be "B2G Emulator" or something? I'm not fully plugged into this, so just throwing out suggestions.
(Assignee)

Comment 15

6 years ago
CCing sheriffs to see if they have a preference in naming.
(Assignee)

Comment 16

6 years ago
Comment on attachment 673447 [details] [diff] [review]
schedule b2g mochitests, rename them

Landing so the ateam has something to work with.
Happy to rename if/when we have something to rename to.

http://hg.mozilla.org/build/buildbot-configs/rev/30ab26725df6
Attachment #673447 - Flags: checked-in+
(Assignee)

Comment 17

6 years ago
This is live.
(Assignee)

Comment 18

6 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #14)
> OK. Are we sure we want them to be called ics_armv7a_gecko? If there's a
> better name we can make TBPL recognize it and sort them correctly. Maybe
> they should be "B2G Emulator" or something? I'm not fully plugged into this,
> so just throwing out suggestions.

Yeah, now they're showing up in the correct row but tbpl thinks they're builds.
So if we're going to need a tbpl patch anyway, I'd be happy to take the ics_armv7a_gecko out of the name.
(In reply to Aki Sasaki [:aki] from comment #18)
> Yeah, now they're showing up in the correct row but tbpl thinks they're
> builds.
> So if we're going to need a tbpl patch anyway, I'd be happy to take the
> ics_armv7a_gecko out of the name.

My TBPL patches (awaiting review) in bug 786314 improve the situation slightly, however the main problem is that the naming of most things B2G is inconsistent

All of the other platforms currently use:
{OS_name} {optional_testing_platform_name_eg_Tegra} {tree_name} {build_or_test_name}

Whereas B2G use an assortment of different string orders and also space vs underscore vs hyphen naming:
b2g_cedar_ics_armv7a_gecko build
b2g_cedar_ics_armv7a_gecko-debug build
b2g_cedar_panda_dep
b2g ics_armv7a_gecko emulator cedar opt test marionette-webapi 

I'm happy to file the necessary bugs to get these straightened out (in both buildbot-configs & TBPL), but first:

1) Are the emulator builds supposed to be shown on the same line as "b2g_{treeName}_ics_armv7a_gecko build"? (As of today, now called "B2G Arm {opt,debug}")

2) Are the new "B2G Arm {opt,debug}" & "B2G Panda opt" rows on TBPL suitably named?

3) If we're wanting to get rid of "ics_armv7a_gecko" from the buildernames, what suggestions are their for its replacement (to differentiate them from panda builds?
(Assignee)

Comment 20

6 years ago
I might not be the best person to answer this, but aiui:

(In reply to Ed Morley (PTO/travelling until 24th Oct) [:edmorley UTC+1] from comment #19)
> 1) Are the emulator builds supposed to be shown on the same line as
> "b2g_{treeName}_ics_armv7a_gecko build"? (As of today, now called "B2G Arm
> {opt,debug}")

Yes.
I would guess we may have additional rows later (e.g. jb_armv7a_gecko for Jelly Bean might be a future row), so we may want to have a way to differentiate.
That isn't an issue currently.

> 2) Are the new "B2G Arm {opt,debug}" & "B2G Panda opt" rows on TBPL suitably
> named?

B2G Panda is fine.

B2G Arm seems fine for now; we may rename if/when we have multiple types of emulator builds.

> 3) If we're wanting to get rid of "ics_armv7a_gecko" from the buildernames,
> what suggestions are their for its replacement (to differentiate them from
> panda builds?

I think ics_armv7a_gecko will stay in the build names, and might go away from the test names.

(marionette-webapi and mochitests-1 through 3 are tests, but show up as builds atm.)
Thank you for the quick reply :-)

(In reply to Aki Sasaki [:aki] from comment #20)
> I think ics_armv7a_gecko will stay in the build names, and might go away
> from the test names.

That would make them different from every other test buildername. We always list the OS/platform in the buildername for tests, since we need a way to associate them with a row on TBPL (and without ics_armv7a_gecko or similar, we'd be left with 'B2G', which could apply to multiple rows).

> (marionette-webapi and mochitests-1 through 3 are tests, but show up as
> builds atm.)

The reason they show up as builds isn't because of the ics_armv7a_gecko specifically, it's because the initial regex to match ics_armv7a_gecko wasn't specific enough, and will be fixed by the patches in bug 786314:
>-        /b2g.*_gecko/i.test(name) ? "B2G Gecko Build" :
>-        /b2g.*_panda_dep/i.test(name) ? "B2G Panda Image" :
>+        /b2g_.*_gecko(?:-debug)? build/i.test(name) ? "B2G Gecko Build" :
>+        /b2g_.*_panda_dep/i.test(name) ? "B2G Panda Image" :

One more question (sorry! :-)) - are we moving towards using underscores in buildernames across the board, or should they be removed from the B2G buildernames to be consistent with everything else?
(Assignee)

Comment 22

6 years ago
(In reply to Ed Morley (PTO/travelling until 24th Oct) [:edmorley UTC+1] from comment #21)
> Thank you for the quick reply :-)

Thank you.
> (In reply to Aki Sasaki [:aki] from comment #20)
> > I think ics_armv7a_gecko will stay in the build names, and might go away
> > from the test names.
> 
> That would make them different from every other test buildername. We always
> list the OS/platform in the buildername for tests, since we need a way to
> associate them with a row on TBPL (and without ics_armv7a_gecko or similar,
> we'd be left with 'B2G', which could apply to multiple rows).

Happy to keep them, then.

> > (marionette-webapi and mochitests-1 through 3 are tests, but show up as
> > builds atm.)
> 
> The reason they show up as builds isn't because of the ics_armv7a_gecko
> specifically, it's because the initial regex to match ics_armv7a_gecko
> wasn't specific enough, and will be fixed by the patches in bug 786314:
> >-        /b2g.*_gecko/i.test(name) ? "B2G Gecko Build" :
> >-        /b2g.*_panda_dep/i.test(name) ? "B2G Panda Image" :
> >+        /b2g_.*_gecko(?:-debug)? build/i.test(name) ? "B2G Gecko Build" :
> >+        /b2g_.*_panda_dep/i.test(name) ? "B2G Panda Image" :

Nice, thanks again.

> One more question (sorry! :-)) - are we moving towards using underscores in
> buildernames across the board, or should they be removed from the B2G
> buildernames to be consistent with everything else?

Hrm, probably the latter.  It may be as simple as tweaking the base_name's in http://hg.mozilla.org/build/buildbot-configs/file/a6af987ce749/mozilla/b2g_config.py to be more like the base_name's in http://hg.mozilla.org/build/buildbot-configs/file/a6af987ce749/mozilla/config.py .
I've made a few tweaks to the TBPL patches in bug 786314 which will work around the buildername variations for now. These have just been pushed to production in bug 805201, so everything on Cedar shows up in TBPL properly now :-)

I'll file a separate bug for sorting on the buildbot side.
Depends on: 786314

Updated

6 years ago
Depends on: 805914
(Assignee)

Comment 24

6 years ago
These are scheduled and bug 805914 is landed.
Reftests and xpcshell are blocked due to external reasons; I can schedule them when they're ready.

I'm going to mark this resolved; please file new bugs for any new issues.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Assignee)

Updated

6 years ago
Blocks: 807125
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.