Closed Bug 940768 Opened 6 years ago Closed 6 years ago

Need helix-eng builds for test automation

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jgriffin, Assigned: bhearsum)

References

Details

Attachments

(3 files)

We're going to start running automated tests on Helix devices soon, and so will need Helix engineering builds on trunk.

We're currently producing Helix user builds; we just need an additional config for the engineering variant (the same as we're currently doing for Hamachi).
Adding mlee, who might be interested in this; one of our motivations for standing up helix automated testing is to be able to support investigation of perf problems if/when reported on the helix device.
Needed for running Camera functional tests.
(In reply to jdarcangelo from comment #2)
> Needed for running Camera functional tests.
For functional testing why can't you build your own? I don't understand why you would need automated builds for functional testing. As much as we'd like to run helix systems in automation, we have no helix systems in our automation rig *at all*. So even if we had the builds, we couldn't use them, so I've asked releng to deprioritize this. 

These are all terrible trade-offs to be making, but we have to make them somewhere. Can you describe why you need automated builds for the camera functional testing you have in mind?  If you don't have capacity to build a helix image, I can look at doing a helix build on my desktop and uploading it some place you can grab it.
Flags: needinfo?(jdarcangelo)
Well, first off, it takes about a half of a working day to build B2G from source on my laptop. Work on the Camera on both the Gaia and Gecko sides is highly active and constantly changing, hence the need for up-to-date builds. Functional testing for the Camera app in Gaia cannot be done via simulator. Additionally, most, if not all, of the new Camera features being worked on are not available on the Hamachi. The Helix has the hardware for these features (front/rear cameras, flash, autofocus, etc.) which is why the functional tests are being run on it. As far as I know, the Python-based functional tests can only run on a device with an ENG build. I was able to eventually finish building B2G on my laptop yesterday and flashed my Helix device and the tests could then be run successfully.

Its more of an inconvenience than anything else. As I stated already, the state of the camera is constantly changing at the present time. So, having to re-build B2G from source on a regular basis will cause a major hit to productivity.
Flags: needinfo?(jdarcangelo)
+1. On having engineering builds for the helix. A bunch of gaia developers rely on the engineering build to keep gecko up to date using the B2G-flash-tool:

https://github.com/Mozilla-TWQA/B2G-flash-tool

We would love the have the same approach across devices. I currently use it for the Hamachi but I cannot with helix.

In the case of the camera we need the eng. build because running functional tests on device is part of our daily workflow to prevent regressions. Camera has features that are hardware dependent and running tests on the simulator is not an option.
Its also worth noting that there is already a Helix build available to the B2G Flash Tool, it just is not the ENG build.
Clint,
Would you happen to know what the build options are for the Hamachi ENG automated builds? I'm seeing strange performance test results with the Helix ENG build I made myself.
Flags: needinfo?(ctalbert)
Assignee: nobody → bhearsum
Are these needed everywhere we already have helix builds?
Ben,
I'm not entirely sure. I think most of us that need it are using the B2G Flash Tool which seems to pull it from here: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/
(In reply to jdarcangelo from comment #7)
> Clint,
> Would you happen to know what the build options are for the Hamachi ENG
> automated builds? I'm seeing strange performance test results with the Helix
> ENG build I made myself.

The Hamache ENG builds are made using this:

10:28:24     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build', '--unpriv', '--shell', u'/usr/bin/env VARIANT=eng GAIA_OPTIMIZE=1 "LESSOPEN=|/usr/bin/lesspipe.sh %s" CVS_RSH=ssh LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld "WGET_OPTS=-c -q" CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20140108085045 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbolpush.mozilla.org PYTHONPATH=:/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/compare-locales/lib L10NBASEDIR=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/gecko-l10n MOZILLA_OFFICIAL=1 "MOZ_CHROME_MULTILOCALE=es-ES pl pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 B2G_SYSTEM_APPS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build HOSTNAME=bld-linux64-ec2-127.build.releng.use1.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000 PROPERTIES_FILE=/builds/slave/b2g_b2g-in_ham_eng_dep-0000000/buildprops.json MAIL=/var/spool/mail/cltbld B2G_UPDATER=1 B2G_UPDATE_CHANNEL=hamachi/default/default ./build.sh buildsymbols'] in /builds/slave/b2g_b2g-in_ham_eng_dep-0000000/build

The most important parts for a local build are VARIANT=eng and GAIA_OPTIMIZE=1
I ran this through staging overnight and it seemed to work, except I screwed up some config somewhere and it failed to upload (logs are at http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-central_helix_eng_nightly/builds/0/steps/run_script/logs/stdio and http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-central_helix_eng_dep/builds/0/steps/run_script/logs/stdio). Because of that I also don't have any build artifacts that someone can verify -- but the logs show that we're setting variant=eng and GAIA_OPTIMIZE=1.

I'm out today, but I wanted to throw this patch up because I know it's high priority. If we want to wait, I can re-do staging builds tomorrow and someone can have a look at them and verify in production.
Attachment #8357817 - Flags: review?(aki)
Comment on attachment 8357817 [details] [diff] [review]
probable buildbot configs

(In reply to Ben Hearsum [:bhearsum] from comment #11)
> Created attachment 8357817 [details] [diff] [review]
> probable buildbot configs
> 
> I ran this through staging overnight and it seemed to work, except I screwed
> up some config somewhere and it failed to upload (logs are at
> http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-
> central_helix_eng_nightly/builds/0/steps/run_script/logs/stdio and
> http://dev-master01.build.scl1.mozilla.com:8018/builders/b2g_mozilla-
> central_helix_eng_dep/builds/0/steps/run_script/logs/stdio). Because of that
> I also don't have any build artifacts that someone can verify -- but the
> logs show that we're setting variant=eng and GAIA_OPTIMIZE=1.

* it's trying to upload to stage.m.o with staging keys, so the upload failure is expected
* artifacts are in the build/upload/ directory

I think this lgtm; thanks!
Attachment #8357817 - Flags: review?(aki) → review+
I think we're done here now...
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
in production
Should I be able to download these ENG builds yet?
This just made it to production today (apologies for the comment on the 10th - I thought it made it there then). I've triggered a mozilla-central helix-eng nightly build right now. They should show up on pvtbuilds in a couple of hours, likely in https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-helix-eng/.

From now on, these builds will run whenever non-eng helix builds run, too.
Thanks Ben!
I'm not sure who should review/merge this patch, but this adds the helix-eng option to the B2G-flash-tool.
Attachment #8359387 - Flags: review?
Flags: needinfo?
Attached patch TBPL supportSplinter Review
In the future, a heads-up about new builds being added is always appreciated.
Attachment #8359577 - Flags: review?(emorley)
Attachment #8359577 - Flags: review?(emorley) → review+
Flags: needinfo?
Attachment #8359387 - Flags: review?
Depends on: 959552
(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #22)
> Comment on attachment 8359577 [details] [diff] [review]
> TBPL support
> 
> https://hg.mozilla.org/webtools/tbpl/rev/1a19aac41258

In production :)
Flags: needinfo?(ctalbert)
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.