841935 - Publish B2G update smoketest flash zips for each build

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Marshall Culpepper [:marshall_law]]

These flash zips are how the b2g update tests reset between each set of tests on updates.

Comment 1

[Marshall Culpepper [:marshall_law]]

Attachment: stage b2g update testdata for upload - v1

Aki, can you test this in a real releng environment? I've only been able to test this by symlinking "build" to my development B2G dir, and only executing my new action, i.e:

./scripts/b2g_build.py \
  --target unagi \
  --repo https://git.mozilla.org/releases/gecko.git \
  --config-file users/marshall/b2g-updates.py \
  --only-build-update-testdata

I'll attach my config next.

Comment 2

[Marshall Culpepper [:marshall_law]]

Attachment: b2g-updates.py (example)

Example config I used to test this. Note you'll need to change smoketest_config.public_key/private_key to point at your copy of the B2G source dir (though, we could do this in code too..)

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 714644 [details] [diff] [review]
stage b2g update testdata for upload - v1

scripts/b2g_build.py:726: local variable 'gecko_config' is assigned to but never used

>+        os.makedirs(upload_testdata_dir)

self.mkdir_p(upload_testdata_dir) , if only for the logging.

>+        for f in ('flash.zip', 'flash.sh'):
>+            self.copy_to_upload_dir(
>+                os.path.join(target_testdata_dir, f),
>+                os.path.join(upload_testdata_dir, f))

This will be problematic because the prep-upload action wipes abs_upload_dir (specifically because depend builds were uploading a lot of cruft).

We could either get this copy into prep_upload(), or we could move the abs_upload_dir clobber into the clobber action, or something similar.

I'll go ahead and run this though; the above issues shouldn't be too hard to fix.

(In reply to Marshall Culpepper [:marshall_law] from comment #2)
> Created attachment 714647 [details]
> b2g-updates.py (example)
> 
> Example config I used to test this. Note you'll need to change
> smoketest_config.public_key/private_key to point at your copy of the B2G
> source dir (though, we could do this in code too..)

I'm going to try os.path.abspath("build/target/...")

Comment 4

[Aki Sasaki (not active)]

Currently looks like I'm going to have time for this later tomorrow; is that ok?

Comment 5

[Aki Sasaki (not active)]

Dying here:
14:44:21     INFO - #####
14:44:21     INFO - ##### Running build-update-testdata step.
14:44:21     INFO - #####
14:44:21     INFO - Running command: ['./build.sh', 'fs_config'] in /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build
14:44:21     INFO - Copy/paste: ./build.sh fs_config
14:44:21    ERROR -  build/envsetup.sh: line 44: make: command not found
14:44:21     INFO -  ** Don't have a product spec for: 'full_unagi'
14:44:21     INFO -  ** Do you have the right repo manifest?
14:44:21     INFO -  
14:44:21     INFO -  > Build failed! <
14:44:21     INFO -  Build with |./build.sh -j1| for better messages
14:44:21     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
14:44:21    ERROR - Return code: 1
14:44:21    FATAL - failed to build fs_config
14:44:21    FATAL - Exiting 2

Landing this fix and retrying.
http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/rev/da46c1a6465e

Comment 6

[Aki Sasaki (not active)]

15:17:46     INFO -  Run |./flash.sh| to flash all partitions of your device
15:17:46     INFO - Return code: 0
15:17:46     INFO - Writing to file /tmp/tmpjKuzEg/smoketest-config.json
15:17:46     INFO - Contents:
15:17:46     INFO -  {"public_key": "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/target/product/security/testkey.x509.pem", "private_key": "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/target/product/security/testkey.pk8", "devices": {"unagi": {"serials": ["full_unagi"], "system_fs_type": "ext4", "sdcard_recovery": "/sdcard", "sdcard": "/mnt/sdcard", "data_fs_type": "ext4", "system_location": "/dev/block/mmcblk0p19", "data_location": "/dev/block/mmcblk0p22"}}}
15:17:46     INFO - Running command: ['/usr/bin/python', '/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py', 'unagi', '/tmp/tmpjKuzEg'] in /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build
15:17:46     INFO - Copy/paste: /usr/bin/python /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py unagi /tmp/tmpjKuzEg
15:17:46     INFO -  Traceback (most recent call last):
15:17:46     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 7, in <module>
15:17:46     INFO -      import argparse
15:17:46     INFO -  ImportError: No module named argparse
15:17:46    ERROR - Return code: 1
15:17:46    FATAL - failed to stage b2g update testdata
15:17:46    FATAL - Exiting 2

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 714644 [details] [diff] [review]
stage b2g update testdata for upload - v1

Minusing for both the upload issue (and nits in comment 3) plus the missing argparse error.

Plus http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/rev/da46c1a6465e .

Comment 8

[Aki Sasaki (not active)]

The problem above seems to be the "/usr/bin/python" in 15:17:46     INFO - Running command: ['/usr/bin/python', '/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py', 'unagi', '/tmp/tmpjKuzEg'] in /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build

This appears to be sys.executable, which is /usr/bin/python, 2.6.6.
We probably need to call via /tools/buildbot/bin/python , which is 2.7.2.
It's possible our staging build slaves are at a lower rev than our production slaves, too.

Comment 9

[Aki Sasaki (not active)]

Turn this on for all update-enabled builds, since we're not sure if we want this for unagi-eng or for more user-type builds.

Comment 10

[Aki Sasaki (not active)]

Ok, just calling with /tools/buildbot/bin/python doesn't work because we call further python scripts down the line that assume we have a 2.7 python first in our PATH.

I think the fix here may be updating the env PATH we run this with.

14:43:58     INFO - Writing to file /tmp/tmp4hzB5T/smoketest-config.json
14:43:58     INFO - Contents:
14:43:58     INFO -  {"public_key": "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/target/product/security/testkey.x509.pem", "private_key": "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/target/product/security/testkey.pk8", "devices": {"unagi": {"serials": ["full_unagi"], "system_fs_type": "ext4", "sdcard_recovery": "/sdcard", "sdcard": "/mnt/sdcard", "data_fs_type": "ext4", "system_location": "/dev/block/mmcblk0p19", "data_location": "/dev/block/mmcblk0p22"}}}
14:43:58     INFO - Running command: ['/tools/buildbot/bin/python', '/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py', 'unagi', '/tmp/tmp4hzB5T'] in /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build
14:43:58     INFO - Copy/paste: /tools/buildbot/bin/python /builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py unagi /tmp/tmp4hzB5T
14:43:59     INFO -  Traceback (most recent call last):
14:43:59     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/tools/update-tools/build-flash-fota.py", line 19, in <module>
14:43:59     INFO -      import argparse
14:43:59     INFO -  ImportError: No module named argparse
14:43:59     INFO -  Building flash zip for device unagi, version 18.0, build 20130221172727...
14:43:59     INFO -  Traceback (most recent call last):
14:43:59     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 95, in <module>
14:43:59     INFO -      main()
14:43:59     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 90, in main
14:43:59     INFO -      stage_update(args.device, args.stage_dir)
14:43:59     INFO -    File "/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 59, in stage_update
14:43:59     INFO -      '--output', flash_zip])
14:43:59     INFO -    File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
14:43:59     INFO -      raise CalledProcessError(retcode, cmd)
14:43:59     INFO -  subprocess.CalledProcessError: Command '['/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/tools/update-tools/build-flash-fota.py', '--system-dir', '/builds/slave/b2g_m-b18_unagi_eng_ntly-00000/build/out/target/product/unagi/system', '--system-fs-type', u'ext4', '--system-location', u'/dev/block/mmcblk0p19', '--data-fs-type', u'ext4', '--data-location', u'/dev/block/mmcblk0p22', '--output', '/tmp/tmp4hzB5T/unagi/20130221172727/flash.zip']' returned non-zero exit status 1
14:43:59    ERROR - Return code: 1

Comment 11

[Aki Sasaki (not active)]

UGH.

Setting python to /tools/python27/bin/python2.7 works for the first call, but the next one doesn't work, even with /tools/python27/bin/ in the PATH, because it's python2.7 not python.

Also, it's not clear to me if I can run this outside of mock, since the entire build directory structure is inside of mock.  So I'd have to change the

testdata_dir = tempfile.mkdtemp()

to write to somewhere accessible from mock.


Marshall -- is it possible to prepend sys.executable in front of the various python calls?  That way if I get the original python call right, we don't have to worry about mucking with PATH and getting python2.7 somehow aliased to python.

Comment 12

[Marshall Culpepper [:marshall_law]]

Attachment: use sys.executable - v1

Aki -- this platform patch should fix the calls to the b2g update tools.

Comment 13

[Aki Sasaki (not active)]

Building flash zip for device unagi, version 18.0, build 20130222184650...
Error: java is required to be on your PATH for signing

We're going to have to add java to the list of packages installed in mock, I think.

Comment 14

[Aki Sasaki (not active)]

http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/b2g/nightly/mozilla-b2g18-unagi-eng/latest/update-testdata/

[ ]	flash.sh	22-Feb-2013 17:28 	572
[ ]	flash.zip	22-Feb-2013 17:28 	38M

Patches incoming.

Comment 15

[Aki Sasaki (not active)]

Attachment: (inbound) add java-1.6.0-openjdk-devel to otoro+unagi mock_packages

This and attachment 717290 [details] [diff] [review] will have to land in m-c, b2g18, and b2g18_v1_0_1.

Comment 16

[Aki Sasaki (not active)]

I suppose we don't need otoro config.json in the above patch, since it looks like we only have a smoketest_config for unagi atm.

Comment 17

[Aki Sasaki (not active)]

Attachment: (mozharness) unagi update smoketest flash zips

This patch:

* adds build-update-testdata to default_actions for all unagi build configs
* adds the python exe to the same configs. Luckily, /tools/python27/bin/python2.7 holds true for both non-mock and mock, although it looks like we're only using this in mock currently.
* adds the smoketest_config for those configs
* adds the build-update-testdata action.  This had to come after the make-updates action, or the scripts would die on the missing mar.
* bases query_abs_dirs() off abs_dirs['abs_work_dir'] since that's already defined
* moves the upload_dir clobber to clobber() so build_update_testdata() can put files in the upload dir
* adds marshall's build_update_testdata(), with mock, python2.7, and checks to make sure we should be running that action

This needs to land after the above gecko patches land and are uplifted to b2g18{,_v1_0_1}.

Comment 18

[Marshall Culpepper [:marshall_law]]

Requesting tef+ so we can get this patch uplifted. I've landed the gecko patch in inbound:
 https://hg.mozilla.org/integration/mozilla-inbound/rev/c863d79841e9

Comment 19

[Chris AtLee [:catlee]]

Comment on attachment 717435 [details] [diff] [review]
(mozharness) unagi update smoketest flash zips

Review of attachment 717435 [details] [diff] [review]:
-----------------------------------------------------------------

Some more comments in build-update-testdata() would be great for future readers of this code.

Comment 20

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/c863d79841e9

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 717411 [details] [diff] [review]
(inbound) add java-1.6.0-openjdk-devel to otoro+unagi mock_packages

https://hg.mozilla.org/integration/mozilla-inbound/rev/0e814444d4b3

Comment 22

[Aki Sasaki (not active)]

Waiting to land the mozharness patch until we've merged the two gecko patches to b2g18 + b2g18_v1_0_1.  Once it's landed we can mark this fixed.

Comment 23

[Aki Sasaki (not active)]

Comment on attachment 717411 [details] [diff] [review]
(inbound) add java-1.6.0-openjdk-devel to otoro+unagi mock_packages

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 

New feature: update testing smoketests.

User impact if declined: 

Lack of automated update testing.

Testing completed: 

Was able to build + upload the smoketest zip in staging.

Risk to taking this patch (and alternatives if risky): 

Low.

String or UUID changes made by this patch:

None.

Comment 24

[Aki Sasaki (not active)]

Comment on attachment 717290 [details] [diff] [review]
use sys.executable - v1

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 

New feature: update testing smoketests.

User impact if declined: 

Lack of automated update testing.

Testing completed: 

Was able to build + upload the smoketest zip in staging.

Risk to taking this patch (and alternatives if risky): 

Low.

String or UUID changes made by this patch:

None.

Comment 25

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/0e814444d4b3

Comment 26

[Lukas Blakk [:lsblakk] use ?needinfo]

Marking blocking so this can be uplifted to both v1-train and v1.0.1 without approval needed.

Comment 27

[Aki Sasaki (not active)]

Landed on b2g18:
http://hg.mozilla.org/releases/mozilla-b2g18/rev/aa38919cb038
http://hg.mozilla.org/releases/mozilla-b2g18/rev/953ede397f25

b2g18_v1_0_1 has some unstarred oranges.

Comment 28

[Aki Sasaki (not active)]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/872b3c951828
https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/14d8e241d9e3

http://hg.mozilla.org/build/mozharness/rev/8015ede70192
Merged to production.

Kicking off some nightlies.

Comment 29

[Aki Sasaki (not active)]

unagi-eng update smoketest zip:
https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18-unagi-eng/latest/update-testdata/

We got this for the unagi nightly:
13:12:16     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g_m-b18_unagi_ntly-000000000/build', '--unpriv', '--shell', u'/usr/bin/env VARIANT=user "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing3.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com CCACHE_DIR=/builds/ccache PYTHONPATH=:/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/compare-locales/lib MOZ_BUILD_DATE=20130225112314 L10NBASEDIR=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko-l10n MOZILLA_OFFICIAL=1 "MOZ_CHROME_MULTILOCALE=es-ES pt-BR" "MOZ_SIGN_CMD=python /builds/slave/b2g_m-b18_unagi_ntly-000000000/scripts/release/signing/signtool.py --cachedir /builds/slave/b2g_m-b18_unagi_ntly-000000000/signing_cache -t /builds/slave/b2g_m-b18_unagi_ntly-000000000/token -n /builds/slave/b2g_m-b18_unagi_ntly-000000000/nonce -c /builds/slave/b2g_m-b18_unagi_ntly-000000000/scripts/release/signing/host.cert -H signing3.srv.releng.scl3.mozilla.com:9100" CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build B2GUPDATER=1 HOSTNAME=bld-linux64-ec2-662.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_m-b18_unagi_ntly-000000000 PROPERTIES_FILE=/builds/slave/b2g_m-b18_unagi_ntly-000000000/buildprops.json MAIL=/var/spool/mail/cltbld /tools/python27/bin/python2.7 /builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko/testing/marionette/update-smoketests/stage-update.py unagi /builds/slave/b2g_m-b18_unagi_ntly-000000000/build/testdata'] in /builds/slave/b2g_m-b18_unagi_ntly-000000000/build
13:12:16     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g_m-b18_unagi_ntly-000000000/build --unpriv --shell "/usr/bin/env VARIANT=user \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_SIGNING_SERVERS=signing3.srv.releng.scl3.mozilla.com:9100 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com CCACHE_DIR=/builds/ccache PYTHONPATH=:/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/compare-locales/lib MOZ_BUILD_DATE=20130225112314 L10NBASEDIR=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko-l10n MOZILLA_OFFICIAL=1 \"MOZ_CHROME_MULTILOCALE=es-ES pt-BR\" \"MOZ_SIGN_CMD=python /builds/slave/b2g_m-b18_unagi_ntly-000000000/scripts/release/signing/signtool.py --cachedir /builds/slave/b2g_m-b18_unagi_ntly-000000000/signing_cache -t /builds/slave/b2g_m-b18_unagi_ntly-000000000/token -n /builds/slave/b2g_m-b18_unagi_ntly-000000000/nonce -c /builds/slave/b2g_m-b18_unagi_ntly-000000000/scripts/release/signing/host.cert -H signing3.srv.releng.scl3.mozilla.com:9100\" CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build B2GUPDATER=1 HOSTNAME=bld-linux64-ec2-662.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g_m-b18_unagi_ntly-000000000 PROPERTIES_FILE=/builds/slave/b2g_m-b18_unagi_ntly-000000000/buildprops.json MAIL=/var/spool/mail/cltbld /tools/python27/bin/python2.7 /builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko/testing/marionette/update-smoketests/stage-update.py unagi /builds/slave/b2g_m-b18_unagi_ntly-000000000/build/testdata"
13:12:17     INFO -  Traceback (most recent call last):
13:12:17     INFO -    File "/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 95, in <module>
13:12:17     INFO -      main()
13:12:17     INFO -    File "/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 90, in main
13:12:17     INFO -      stage_update(args.device, args.stage_dir)
13:12:17     INFO -    File "/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko/testing/marionette/update-smoketests/stage-update.py", line 27, in stage_update
13:12:17     INFO -      b2g_config = b2g.import_update_tools().B2GConfig()
13:12:17     INFO -    File "/builds/slave/b2g_m-b18_unagi_ntly-000000000/build/gecko/testing/marionette/client/marionette/b2ginstance.py", line 94, in import_update_tools
13:12:17     INFO -      import update_tools
13:12:17     INFO -  ImportError: No module named update_tools
13:12:17    ERROR - Return code: 1
13:12:17    FATAL - failed to stage b2g update testdata
13:12:17    FATAL - Exiting 2

RyanVM re-kicked it; hopefully it's a one time glitch =\

Comment 30

[Aki Sasaki (not active)]

Nope, it's a real problem.

Backed out 8015ede70192: http://hg.mozilla.org/build/mozharness/rev/301d66af8e86

We need to figure out:

a) do we need this on unagi-beta as well as unagi-eng, and if so
b) what's update_tools and why can't we import it in unagi-beta as opposed to unagi-eng

Comment 31

[Aki Sasaki (not active)]

c) if someone could verify https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18-unagi-eng/2013/02/2013-02-25-11-23-14/update-testdata/ , that would be awesome as well.

Comment 32

[Aki Sasaki (not active)]

Asking Marshall for info on comment 30 question (b).

Comment 33

[Aki Sasaki (not active)]

Asking Stephen for info on comment 30 question (a) (I think this is a yes) and comment 31.

Comment 34

[Aki Sasaki (not active)]

Looks like we need new snapshots for unagi + unagi-stable to proceed.

Comment 35

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Aki Sasaki [:aki] from comment #33)
> Asking Stephen for info on comment 30 question (a) (I think this is a yes)
> and comment 31.

Sorry to pass the buck, but I'm not actually sure; I've always been going off https://bugzilla.mozilla.org/show_bug.cgi?id=826450#c2 and its associated links, so I think (Marshall, please correct me!) for now we'll target unagi engineering builds (because of the need to use Marionette), but once we get that working, and if we have the infra, I'd be happy to target beta -> beta updates, too.

Comment 36

[Aki Sasaki (not active)]

(In reply to Stephen Donner [:stephend] from comment #35)
> (In reply to Aki Sasaki [:aki] from comment #33)
> > Asking Stephen for info on comment 30 question (a) (I think this is a yes)
> > and comment 31.
> 
> Sorry to pass the buck, but I'm not actually sure; I've always been going
> off https://bugzilla.mozilla.org/show_bug.cgi?id=826450#c2 and its
> associated links, so I think (Marshall, please correct me!) for now we'll
> target unagi engineering builds (because of the need to use Marionette), but
> once we get that working, and if we have the infra, I'd be happy to target
> beta -> beta updates, too.

unagi-eng is what's working atm, so we can turn this back on if we limit to just unagi-eng.  Or we can wait til we resolve bug 845000.

Does the testdata zip look good?

Comment 37

[Marshall Culpepper [:marshall_law]]

Attachment: flash diff

(In reply to Aki Sasaki [:aki] from comment #31)
> c) if someone could verify
> https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18-
> unagi-eng/2013/02/2013-02-25-11-23-14/update-testdata/ , that would be
> awesome as well.

This is a quick comparison with a flash.zip I have from last month. It looks like the uploaded zip is missing at the least "webapps" directory, and some fonts that might be required for gaia to run. Any idea why the webapps dir might be missing from the out/target/product/unagi/system/b2g dir?

I tried flashing this and it's definitely not suitable for running tests :)

Comment 38

[Aki Sasaki (not active)]

=\

I'll try again when all the snapshots are updated.  Hopefully that's tomorrow.

Comment 39

[Aki Sasaki (not active)]

We still don't have update_tools for unagi after the snapshot update.
Should I enable for just unagi-eng, or continue to debug, or both?

Comment 40

[Jonathan Griffin (:jgriffin)]

Since we need these for tests that require Marionette, I would say unagi-eng is enough, unless marshall says otherwise.

Comment 41

[Aki Sasaki (not active)]

Comment on attachment 717435 [details] [diff] [review]
(mozharness) unagi update smoketest flash zips

Relanded, without the releng or releng-beta configs: http://hg.mozilla.org/build/mozharness/rev/af724a45b16b
Merged to production.

We should get unagi-eng smoketest flash zips again without the unagi bustage.  
Kicking off some nightlies to verify.

If we still have issues with the zips themselves, we can address those issues while still leaving this patch landed.

Comment 42

[Aki Sasaki (not active)]

Ok, we've got the flash zips for unagi-eng again:

https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18-unagi-eng/latest/update-testdata/
https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18_v1_0_1-unagi-eng/latest/update-testdata/

However, they both appear to be missing the webapps directory, per comment 37.

(In reply to Marshall Culpepper [:marshall_law] from comment #37)
> This is a quick comparison with a flash.zip I have from last month. It looks
> like the uploaded zip is missing at the least "webapps" directory, and some
> fonts that might be required for gaia to run. Any idea why the webapps dir
> might be missing from the out/target/product/unagi/system/b2g dir?

Looking at https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18_v1_0_1-unagi-eng/2013/02/2013-02-27-07-02-00/logs/log_info.log , and searching for 'webapps', I don't really see it in the system/b2g/ directory ever.

I do see

12:38:15     INFO -  rm -rf out/target/product/unagi/data/local/webapps

and some references to webapps in the multilocale portion of the build.

As to why, not really; this seems to be in the murky b2g build system that I have very little familiarity with.

Comment 43

[Aki Sasaki (not active)]

Where does the system/b2g/webapps directory come from?
Is that on master + v1-train + v1.0.1 of whatever repo?

Comment 44

[Marshall Culpepper [:marshall_law]]

(In reply to Aki Sasaki [:aki] from comment #43)
> Where does the system/b2g/webapps directory come from?
> Is that on master + v1-train + v1.0.1 of whatever repo?

Ahh -- the fact that we're using the "eng" build variant might be the root of the problem. Is it possible for us to change to userdebug? Marionette and all the rest should still be enabled there (I test with it here a lot). Without changing to userdebug, it looks like we *might* be able to squeeze by with B2G_SYSTEM_APPS=1.. see this block of code:

https://github.com/mozilla-b2g/gaia/blob/master/Android.mk#L22-32

Barring those options, we might need another build patch :(

Comment 45

[Aki Sasaki (not active)]

I don't know the history of calling this eng vs userdebug, so I don't know what will happen if we change.  I should be able to add B2G_SYSTEM_APPS=1 though.

Comment 46

[Aki Sasaki (not active)]

Attachment: b2g_system_apps

Comment 47

[Aki Sasaki (not active)]

Comment on attachment 719803 [details] [diff] [review]
b2g_system_apps

http://hg.mozilla.org/build/mozharness/rev/287605cb149a
Merged to production.

I need to kick off nightlies for bug 843303 anyway; doing so.

Comment 48

[Aki Sasaki (not active)]

I see system/b2g/webapps/...

https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-b2g18-unagi-eng/2013/03/2013-03-01-12-00-39/update-testdata/flash.zip

Are we done here or is there something else still broken?

Comment 49

[Marshall Culpepper [:marshall_law]]

Ack, apologies, I missed this :) Downloading / flashing this now..

Comment 50

[Marshall Culpepper [:marshall_law]]

Yup, this is working now on my unagi. Thanks again for all your help Aki :)

Comment 51

[Aki Sasaki (not active)]

Awesome.