940768 - Need helix-eng builds for test automation

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

We're going to start running automated tests on Helix devices soon, and so will need Helix engineering builds on trunk.

We're currently producing Helix user builds; we just need an additional config for the engineering variant (the same as we're currently doing for Hamachi).

Comment 1

[Jonathan Griffin (:jgriffin)]

Adding mlee, who might be interested in this; one of our motivations for standing up helix automated testing is to be able to support investigation of perf problems if/when reported on the helix device.

Comment 2

[Justin D'Arcangelo [:justindarc]]

Needed for running Camera functional tests.

Comment 3

[cmtalbert]

(In reply to jdarcangelo from comment #2)
> Needed for running Camera functional tests.
For functional testing why can't you build your own? I don't understand why you would need automated builds for functional testing. As much as we'd like to run helix systems in automation, we have no helix systems in our automation rig *at all*. So even if we had the builds, we couldn't use them, so I've asked releng to deprioritize this. 

These are all terrible trade-offs to be making, but we have to make them somewhere. Can you describe why you need automated builds for the camera functional testing you have in mind?  If you don't have capacity to build a helix image, I can look at doing a helix build on my desktop and uploading it some place you can grab it.

Comment 4

[Justin D'Arcangelo [:justindarc]]

Well, first off, it takes about a half of a working day to build B2G from source on my laptop. Work on the Camera on both the Gaia and Gecko sides is highly active and constantly changing, hence the need for up-to-date builds. Functional testing for the Camera app in Gaia cannot be done via simulator. Additionally, most, if not all, of the new Camera features being worked on are not available on the Hamachi. The Helix has the hardware for these features (front/rear cameras, flash, autofocus, etc.) which is why the functional tests are being run on it. As far as I know, the Python-based functional tests can only run on a device with an ENG build. I was able to eventually finish building B2G on my laptop yesterday and flashed my Helix device and the tests could then be run successfully.

Its more of an inconvenience than anything else. As I stated already, the state of the camera is constantly changing at the present time. So, having to re-build B2G from source on a regular basis will cause a major hit to productivity.

Comment 5

[Diego Marcos [:dmarcos]]

+1. On having engineering builds for the helix. A bunch of gaia developers rely on the engineering build to keep gecko up to date using the B2G-flash-tool:

https://github.com/Mozilla-TWQA/B2G-flash-tool

We would love the have the same approach across devices. I currently use it for the Hamachi but I cannot with helix.

In the case of the camera we need the eng. build because running functional tests on device is part of our daily workflow to prevent regressions. Camera has features that are hardware dependent and running tests on the simulator is not an option.

Comment 6

[Justin D'Arcangelo [:justindarc]]

Its also worth noting that there is already a Helix build available to the B2G Flash Tool, it just is not the ENG build.

Comment 7

[Justin D'Arcangelo [:justindarc]]

Clint,
Would you happen to know what the build options are for the Hamachi ENG automated builds? I'm seeing strange performance test results with the Helix ENG build I made myself.

Comment 8

[[e-mail] (:bhearsum)]

Are these needed everywhere we already have helix builds?

Comment 9

[Justin D'Arcangelo [:justindarc]]

Ben,
I'm not entirely sure. I think most of us that need it are using the B2G Flash Tool which seems to pull it from here: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/

Comment 10

[Jonathan Griffin (:jgriffin)]

(In reply to jdarcangelo from comment #7)
> Clint,
> Would you happen to know what the build options are for the Hamachi ENG
> automated builds? I'm seeing strange performance test results with the Helix
> ENG build I made myself.

The Hamache ENG builds are made using this:

10:28:24     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build', '--unpriv', '--shell', u'/usr/bin/env VARIANT=eng GAIA_OPTIMIZE=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" CVS_RSH=ssh LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld "WGET_OPTS=-c -q" CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20140108085045 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/gecko-l10n MOZILLA_OFFICIAL=1 "MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build HOSTNAME=bld-linux64-ec2-127.build.releng.use1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000 PROPERTIES_FILE=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 B2G_UPDATE_CHANNEL=hamachi/default/default ./build.sh buildsymbols'] in /builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build

The most important parts for a local build are VARIANT=eng and GAIA_OPTIMIZE=1

Comment 11

[[e-mail] (:bhearsum)]

Attachment: probable buildbot configs

I ran this through staging overnight and it seemed to work, except I screwed up some config somewhere and it failed to upload (logs are at http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-central_helix_eng_nightly/builds/0/steps/run_script/logs/stdio and http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-central_helix_eng_dep/builds/0/steps/run_script/logs/stdio). Because of that I also don't have any build artifacts that someone can verify -- but the logs show that we're setting variant=eng and GAIA_OPTIMIZE=1.

I'm out today, but I wanted to throw this patch up because I know it's high priority. If we want to wait, I can re-do staging builds tomorrow and someone can have a look at them and verify in production.

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 8357817 [details] [diff] [review]
probable buildbot configs

(In reply to Ben Hearsum [:bhearsum] from comment #11)
> Created attachment 8357817 [details] [diff] [review]
> probable buildbot configs
> 
> I ran this through staging overnight and it seemed to work, except I screwed
> up some config somewhere and it failed to upload (logs are at
> http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-
> central_helix_eng_nightly/builds/0/steps/run_script/logs/stdio and
> http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-
> central_helix_eng_dep/builds/0/steps/run_script/logs/stdio). Because of that
> I also don't have any build artifacts that someone can verify -- but the
> logs show that we're setting variant=eng and GAIA_OPTIMIZE=1.

* it's trying to upload to stage.m.o with staging keys, so the upload failure is expected
* artifacts are in the build/upload/ directory

I think this lgtm; thanks!

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 8357817 [details] [diff] [review]
probable buildbot configs

https://hg.mozilla.org/build/buildbot-configs/rev/2261ce801c35

Comment 14

[Aki Sasaki (not active)]

Followup fix for test-masters.sh:
https://hg.mozilla.org/build/buildbot-configs/rev/db82327c66d6

Comment 15

[[e-mail] (:bhearsum)]

I think we're done here now...

Comment 16

[Rail Aliiev [:rail]]

in production

Comment 17

[Justin D'Arcangelo [:justindarc]]

Should I be able to download these ENG builds yet?

Comment 18

[[e-mail] (:bhearsum)]

This just made it to production today (apologies for the comment on the 10th - I thought it made it there then). I've triggered a mozilla-central helix-eng nightly build right now. They should show up on pvtbuilds in a couple of hours, likely in https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-helix-eng/.

From now on, these builds will run whenever non-eng helix builds run, too.

Comment 19

[Justin D'Arcangelo [:justindarc]]

Thanks Ben!

Comment 20

[Justin D'Arcangelo [:justindarc]]

Attachment: Patch to add helix-eng option to B2G-flash-tool

I'm not sure who should review/merge this patch, but this adds the helix-eng option to the B2G-flash-tool.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Attachment: TBPL support

In the future, a heads-up about new builds being added is always appreciated.

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8359577 [details] [diff] [review]
TBPL support

https://hg.mozilla.org/webtools/tbpl/rev/1a19aac41258

Comment 23

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #22)
> Comment on attachment 8359577 [details] [diff] [review]
> TBPL support
> 
> https://hg.mozilla.org/webtools/tbpl/rev/1a19aac41258

In production :)