872765 - Schedule all emulator tests on cedar using new full-stack emulator build

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We now have full-stack emulator builds being produced by buildbot.  We should turn on all the tests for this build on cedar, so we can compare against the current emulator build.  If they produce equivalent results, we can move globally to the new build.

Comment 1

[Aki Sasaki (not active)]

This is tricky, since we have two inputs: the build+tests from one build, and the emulator from another build.

Do we just grab the emulator.tar.gz from, e.g. https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-generic/20130528082822/ , in place of the tooltool download?  (is that the main change?)

The easiest thing would probably be to grab the latest emulator on any given test run, but that seems prone to failure, especially on try or retriggering previous tests.  There's also a race condition.

Comment 2

[Jonathan Griffin (:jgriffin)]

> This is tricky, since we have two inputs: the build+tests from one build,
> and the emulator from another build.

Can we generate tests.zip from the emulator build?  This seems preferable, then we won't even need the current ics_armv7a_gecko builds, I don't believe.

The other thing we'll have to do is modify the mozharness scripts not to pass --gecko-path to the runners.  The new full-stack emulator builds have the correct version of gecko built into them, so there's no need to install a different copy.

Currently, the mozharness scripts are more or less hardcoded to assume that self.binary_path needs to be passed as --gecko-path.  We might be able to use ahal's work in bug 872164 to conditionally change this behavior per-tree.

Comment 3

[Jonathan Griffin (:jgriffin)]

> Can we generate tests.zip from the emulator build?  This seems preferable, then 
> we won't even need the current ics_armv7a_gecko builds, I don't believe.

I'm guessing that in order to make this easy, we'll need to change ./build.sh package-tests to generate the necessary tests.zip package.

Comment 4

[Aki Sasaki (not active)]

Attachment: add package-tests to the build.sh targets for emulator

Comment 5

[Aki Sasaki (not active)]

Comment on attachment 754937 [details] [diff] [review]
add package-tests to the build.sh targets for emulator

Hm, should I add the equivalent of

        "{workdir}/out/target/product/panda/*.tar.bz2",                            
        "{workdir}/out/target/product/panda/tests/*.zip",

to the upload_files?

Would the product be 'generic' ?

Comment 6

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #5)
> Comment on attachment 754937 [details] [diff] [review]
> add package-tests to the build.sh targets for emulator
> 
> Hm, should I add the equivalent of
> 
>         "{workdir}/out/target/product/panda/*.tar.bz2",                     
> 
>         "{workdir}/out/target/product/panda/tests/*.zip",
> 
> to the upload_files?
> 
> Would the product be 'generic' ?

Yes, and yes.

Comment 7

[Aki Sasaki (not active)]

Attachment: also upload

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 754956 [details] [diff] [review]
also upload

Review of attachment 754956 [details] [diff] [review]:
-----------------------------------------------------------------

There are currently no *.tar.bz2 files at "{workdir}/out/target/product/generic/*.tar.bz2", but it's nice future-proofing to include this, as long as the lack of files there won't cause any errors at upload time.

Comment 9

[Aki Sasaki (not active)]

Hm.
Verifying: https://tbpl.mozilla.org/?tree=Try&rev=a3fb12ca1f99

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 754956 [details] [diff] [review]
also upload

That went green on try.
https://hg.mozilla.org/integration/mozilla-inbound/rev/d3a809d08cc9

Comment 11

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/d3a809d08cc9

Comment 12

[Andrew Halberstadt [:ahal]]

Are these builds the "mozilla-*-generic" ones on pvtbuilds? If so I'm seeing gaia-tests.zip there but no tests.zip (i.e the regular unittests).

Comment 13

[Aki Sasaki (not active)]

Me too. I'm wondering if that's dependent on bug 876818.

Comment 14

[Jonathan Griffin (:jgriffin)]

Yes, it's dependent on bug 876818.

Comment 15

[Aki Sasaki (not active)]

Woot, https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-generic/20130603144916/b2g-24.0a1.en-US.android-arm.tests.zip

Now for the sendchange.

Comment 16

[Aki Sasaki (not active)]

Attachment: emulator sendchange

Sendchange from the full emulator builds.

Comment 17

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #15)
> Woot,
> https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-
> central-generic/20130603144916/b2g-24.0a1.en-US.android-arm.tests.zip
> 
> Now for the sendchange.

\o/

Comment 18

[Aki Sasaki (not active)]

Comment on attachment 757751 [details] [diff] [review]
emulator sendchange

https://hg.mozilla.org/build/mozharness/rev/512c3dd17546
Merged to production.

Comment 19

[Chris AtLee [:catlee]]

Once this is live on all branches, I believe we can shut off the "B2G Arm opt" builds.

How about debug? Should we do debug emulator builds?

Comment 20

[Aki Sasaki (not active)]

I would guess yes, debug emu builds.

Comment 21

[Jonathan Griffin (:jgriffin)]

Yes, we're running debug emulator tests on cedar and mozilla-b2g18.

Comment 22

[Aki Sasaki (not active)]

Copy/paste: /tools/buildbot/bin/buildbot sendchange --master buildbot-master36.build.mozilla.org:9301 --username sendchange-unittest --branch mozilla-inbound-b2g_emulator-opt-unittest -r c82c44986cc7e3900d7752acd3c2e0f9e645d317 --username [e-mail] --comments "bug 873622 - remove nsScriptSecurityManager::sXPConnect r=bholley" --property builduid:cc28f1276c434817aa89b54c9b052d46 --property buildid:20130604165417 --property pgo_build:False http://pvtbuilds.pvt.build.mozilla.org//pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-generic/20130604165417/emulator.tar.gz http://pvtbuilds.pvt.build.mozilla.org//pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-generic/20130604165417/b2g-24.0a1.en-US.android-arm.tests.zip

Awesome.

Comment 23

[Aki Sasaki (not active)]

Attachment: create emulator testers on cedar with --no-update

This also creates debug testers, even though there aren't yet debug builders.

Comment 24

[Aki Sasaki (not active)]

Attachment: production-masters.json update

Comment 25

[Rail Aliiev [:rail]]

I'll review the patches tomorrow, but from I see now I can say that you still need puppet patches for the new platforms (for masters).

Comment 26

[Aki Sasaki (not active)]

It looks like we're getting the ics_armv7_gecko builds debug via mozconfig: |ac_add_options --enable-debug|

What's the equivalent in build.sh ?  Afaict there are no mozconfigs involved.

Comment 27

[Aki Sasaki (not active)]

Attachment: puppet

Comment 28

[Aki Sasaki (not active)]

Attachment: puppet-manifests

good catch.

Comment 29

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #26)
> It looks like we're getting the ics_armv7_gecko builds debug via mozconfig:
> |ac_add_options --enable-debug|
> 
> What's the equivalent in build.sh ?  Afaict there are no mozconfigs involved.

build.sh uses a mozconfig from gonk-misc/default-gecko-config.  If B2G_DEBUG is defined in the build's environment, it will add --enable-debug to the config.  See default-gecko-config for other relevant variables.

Comment 30

[Jonathan Griffin (:jgriffin)]

quick link: https://github.com/mozilla-b2g/gonk-misc/blob/master/default-gecko-config

Comment 31

[Rail Aliiev [:rail]]

Comment on attachment 762428 [details] [diff] [review]
puppet-manifests

IIRC, we switched to PuppetAgain on all related masters, but it won't hurt.

Comment 32

[Aki Sasaki (not active)]

Comment on attachment 762428 [details] [diff] [review]
puppet-manifests

http://hg.mozilla.org/build/puppet-manifests/rev/a7c2d2c521f6

Comment 33

[Aki Sasaki (not active)]

Comment on attachment 762427 [details] [diff] [review]
puppet

http://hg.mozilla.org/build/puppet/rev/e311c1b1a2c4

Comment 34

[Aki Sasaki (not active)]

Comment on attachment 762386 [details] [diff] [review]
production-masters.json update

http://hg.mozilla.org/build/tools/rev/b4a1c2549daa

Comment 35

[Aki Sasaki (not active)]

Comment on attachment 762385 [details] [diff] [review]
create emulator testers on cedar with --no-update

http://hg.mozilla.org/build/buildbot-configs/rev/4baa1e984601

Comment 36

[Aki Sasaki (not active)]

This is in production.

Comment 37

[Aki Sasaki (not active)]

Attachment: (configs) [needs testing] debug emulator builds

Comment 38

[Aki Sasaki (not active)]

Attachment: (mozharness) --debug and B2G_DEBUG

Comment 39

[Aki Sasaki (not active)]

Attachment: (configs) debug emulator builds

Comment 40

[Aki Sasaki (not active)]

Comment on attachment 763887 [details] [diff] [review]
(mozharness) --debug and B2G_DEBUG

https://hg.mozilla.org/build/mozharness/rev/0be5b453d885

Comment 41

[Aki Sasaki (not active)]

Comment on attachment 764001 [details] [diff] [review]
(configs) debug emulator builds

https://hg.mozilla.org/build/buildbot-configs/rev/120daaccfeda

Comment 42

[Ed Morley [:emorley]]

Now that both parts of bug 880084 have landed, I retriggered a cedar job, but get:
https://tbpl.mozilla.org/php/getParsedLog.php?id=24326891&tree=Cedar
{
02:54:59     INFO - #####
02:54:59     INFO - ##### Running install step.
02:54:59     INFO - #####
02:54:59     INFO - Getting output from command: ['/home/cltbld/talos-slave/test/build/venv/bin/pip', 'freeze']
02:54:59     INFO - Copy/paste: /home/cltbld/talos-slave/test/build/venv/bin/pip freeze
02:54:59     INFO - Reading from file tmpfile_stdout
02:54:59     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
02:54:59     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
02:54:59     INFO - Detecting whether we're running mozinstall >=1.0...
02:54:59     INFO - Getting output from command: ['/home/cltbld/talos-slave/test/build/venv/bin/mozinstall', '--app', 'b2g', '-h']
02:54:59     INFO - Copy/paste: /home/cltbld/talos-slave/test/build/venv/bin/mozinstall --app b2g -h
02:54:59     INFO - Reading from file tmpfile_stdout
02:54:59     INFO - Output received:
02:54:59     INFO -  Usage: mozinstall [options] installer
02:54:59     INFO -  Options:
02:54:59     INFO -    -h, --help            show this help message and exit
02:54:59     INFO -    -d DEST, --destination=DEST
02:54:59     INFO -                          Directory to install application into. [default:
02:54:59     INFO -                          "/home/cltbld/talos-slave/test"]
02:54:59     INFO -    --app=APP             Application being installed. [default: firefox]
02:54:59     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
02:54:59     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
02:54:59     INFO - mkdir: /home/cltbld/talos-slave/test/build/application
02:54:59     INFO - Getting output from command: ['/home/cltbld/talos-slave/test/build/venv/bin/mozinstall', '--app', 'b2g', '/home/cltbld/talos-slave/test/build/emulator.tar.gz', '--destination', '/home/cltbld/talos-slave/test/build/application']
02:54:59     INFO - Copy/paste: /home/cltbld/talos-slave/test/build/venv/bin/mozinstall --app b2g /home/cltbld/talos-slave/test/build/emulator.tar.gz --destination /home/cltbld/talos-slave/test/build/application
02:55:07    ERROR - Errors received:
02:55:07     INFO - Reading from file tmpfile_stderr
02:55:07    ERROR -  Traceback (most recent call last):
02:55:07    ERROR -    File "/home/cltbld/talos-slave/test/build/venv/bin/mozinstall", line 9, in <module>
02:55:07    ERROR -      load_entry_point('mozInstall==1.6', 'console_scripts', 'mozinstall')()
02:55:07    ERROR -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/mozinstall/mozinstall.py", line 300, in install_cli
02:55:07    ERROR -      binary = get_binary(install_path, app_name=options.app)
02:55:07    ERROR -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/mozinstall/mozinstall.py", line 87, in get_binary
02:55:07    ERROR -      raise InvalidBinary('"%s" does not contain a valid binary.' % path)
02:55:07    ERROR -  mozinstall.mozinstall.InvalidBinary: "/home/cltbld/talos-slave/test/build/application/b2g-distro" does not contain a valid binary.
02:55:07     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1
02:55:07     INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1
02:55:07    ERROR - Return code: 1
02:55:07    FATAL - Halting on failure while running ['/home/cltbld/talos-slave/test/build/venv/bin/mozinstall', '--app', 'b2g', '/home/cltbld/talos-slave/test/build/emulator.tar.gz', '--destination', '/home/cltbld/talos-slave/test/build/application']
}

Comment 43

[Andrew Halberstadt [:ahal]]

I filed bug 884828 for this. We could fix this via mozharness config, but I believe mozinstall could use a bit of love anyway.

Comment 44

[Jonathan Griffin (:jgriffin)]

We can fix this directly in the mozharness script.  The root of the issue is that emulator.tar.gz doesn't need to be "mozinstalled", it just needs to be extracted.  Do we want mozinstall to handle this case natively?

In either case, I'll patch the mozharness script today, since that's a faster fix.

Comment 45

[Jonathan Griffin (:jgriffin)]

Attachment: Don't call mozinstall for the emulator,

Tested locally this time.

Comment 46

[Jonathan Griffin (:jgriffin)]

Didn't mean to re-assign this; bzexport did it automatically.

Comment 47

[Andrew Halberstadt [:ahal]]

Comment on attachment 764841 [details] [diff] [review]
Don't call mozinstall for the emulator,

Review of attachment 764841 [details] [diff] [review]:
-----------------------------------------------------------------

I didn't realize there was a separate config for "update runs"

Comment 48

[Jonathan Griffin (:jgriffin)]

Comment on attachment 764841 [details] [diff] [review]
Don't call mozinstall for the emulator,

https://hg.mozilla.org/build/mozharness/rev/f36fa35e6638

Comment 49

[Aki Sasaki (not active)]

The mozharness portions have been merged to production.

Comment 50

[Andrew Halberstadt [:ahal]]

There's a new failure because we are still specifying --gecko-path on the command line even for non update builds: https://hg.mozilla.org/build/mozharness/file/2cc1e4faa57d/configs/b2g/emulator_automation_config.py#l44

I think it might be acceptable to remove --gecko-path from the config file and only append it for update tests in the script. Doing so will mean we can't do custom try pushes with it, but I don't think anyone would want to anyway.

Comment 51

[Jonathan Griffin (:jgriffin)]

We'll want to retain that ability in the short term, so we can run try runs using a pre-built x86 emulator (bug 753928).  I'll have another pack for the gecko_path problem shortly.

Comment 52

[Jonathan Griffin (:jgriffin)]

Attachment: Strip 'gecko-path' from test options if not installing a new gecko,

Comment 53

[Andrew Halberstadt [:ahal]]

Comment on attachment 764890 [details] [diff] [review]
Strip 'gecko-path' from test options if not installing a new gecko,

Review of attachment 764890 [details] [diff] [review]:
-----------------------------------------------------------------

I guess this is fine. Wouldn't you still be able to do try pushes with an x86 emulator though? The mozharness script should still set up --gecko-path properly, the only thing you wouldn't be able to do is specify a *custom* --gecko-path

Comment 54

[Andrew Halberstadt [:ahal]]

Attachment: only append --gecko-path if update_files is set

Per irc discussion jgriffin and I decided this would be a bit cleaner

Comment 55

[Andrew Halberstadt [:ahal]]

Comment on attachment 764900 [details] [diff] [review]
only append --gecko-path if update_files is set

https://hg.mozilla.org/build/mozharness/rev/d06bffd0b1eb

Comment 56

[Andrew Halberstadt [:ahal]]

There's another problem now that doesn't look mozharness related. I think we should resolve this bug and file new bugs for the next set of issues we come across. Feel free to re-open if you disagree.

Comment 57

[Aki Sasaki (not active)]

There was a reconfig, so the debug builders should show up now.