838261 - B2G panda builds need to include download-panda bits

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

This is hitting us on all branches that build b2g. Originally this was thought to be related to the earlier scl3 issues, but we're still seeing failing builds after that was resolved.

https://tbpl.mozilla.org/php/getParsedLog.php?id=19459089&tree=Mozilla-Inbound

10:26:27     INFO -  pulling from http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central
10:26:27     INFO -  searching for changes
10:26:27     INFO -  adding changesets
10:26:27     INFO -  adding manifests
10:26:27     INFO -  adding file changes
10:26:27     INFO -  added 30 changesets with 109 changes to 96 files
10:26:27    ERROR -  abort: HTTP Error 500: Internal Server Error
10:26:27    ERROR -  Automation Error: hg not responding
10:26:27     INFO -  command: ERROR
10:26:27     INFO -  Traceback (most recent call last):
10:26:27     INFO -    File "<string>", line 40, in run_cmd
10:26:27     INFO -    File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
10:26:27     INFO -      raise CalledProcessError(retcode, cmd)
10:26:27     INFO -  CalledProcessError: Command '['hg', 'pull', 'http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central']' returned non-zero exit status 255
10:26:27     INFO -  command: END (15.56s elapsed)
10:26:27     INFO -  Problem pulling from mirror http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central
10:26:27     INFO -  Traceback (most recent call last):
10:26:27     INFO -    File "<string>", line 224, in pull
10:26:27     INFO -    File "<string>", line 246, in pull
10:26:27     INFO -    File "<string>", line 40, in run_cmd
10:26:27     INFO -    File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
10:26:27     INFO -      raise CalledProcessError(retcode, cmd)
10:26:27     INFO -  CalledProcessError: Command '['hg', 'pull', 'http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central']' returned non-zero exit status 255
10:26:27     INFO -  Pulling from mirrors failed; falling back to http://hg.mozilla.org/integration/gaia-central


and later


10:29:46     INFO -  Blob setup script has changed, re-running
10:29:46     INFO -  shasum: imgtec-panda-imm76i-67545da7.tgz: No such file or directory
10:29:46     INFO -  shasum: WARNING: 1 of 1 listed files could not be read
10:29:46     INFO -  imgtec-panda-imm76i-67545da7.tgz: FAILED open or read
10:29:46     INFO -    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
10:29:46     INFO -                                   Dload  Upload   Total   Spent    Left  Speed
10:29:47     INFO -  
10:29:47     INFO -   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
10:29:47     INFO - 108   869  108   869    0     0   1543      0 --:--:-- --:--:-- --:--:--  7427
10:29:47     INFO -  shasum: WARNING: 1 of 1 computed checksums did NOT match
10:29:47     INFO -  imgtec-panda-imm76i-67545da7.tgz: FAILED
10:29:47     INFO -  
10:29:47     INFO -  > Build failed! <
10:29:47     INFO -  Build with |./build.sh -j1| for better messages
10:29:47     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
10:29:47    ERROR - Return code: 1
10:29:47     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g-m-in-panda-dep/build', '--unpriv', '--shell', u'/usr/bin/env "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g-m-in-panda-dep/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130205102120 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g-m-in-panda-dep/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com PYTHONPATH=:/builds/slave/b2g-m-in-panda-dep/compare-locales/lib L10NBASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gecko-l10n "MOZ_CHROME_MULTILOCALE=es-ES pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-625.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json MAIL=/var/spool/mail/cltbld ccache -s'] in /builds/slave/b2g-m-in-panda-dep/build
10:29:47     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g-m-in-panda-dep/build --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g-m-in-panda-dep/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130205102120 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g-m-in-panda-dep/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com PYTHONPATH=:/builds/slave/b2g-m-in-panda-dep/compare-locales/lib L10NBASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gecko-l10n \"MOZ_CHROME_MULTILOCALE=es-ES pt-BR\" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-625.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json MAIL=/var/spool/mail/cltbld ccache -s"
10:29:47     INFO -  cache directory                     /builds/ccache
10:29:47     INFO -  cache hit (direct)                     0
10:29:47     INFO -  cache hit (preprocessed)               0
10:29:47     INFO -  cache miss                             0
10:29:47     INFO -  files in cache                    229449
10:29:47     INFO -  cache size                           9.3 Gbytes
10:29:47     INFO -  max cache size                      10.0 Gbytes
10:29:48     INFO - Return code: 0
10:29:48    FATAL - failed to build
10:29:48    FATAL - Exiting 2

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Forgot to note that m-c, inbound, b2g18, and b2g18_v1_0_0 are all closed until this is resolved.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage.

Comment 3

[Aki Sasaki (not active)]

https://github.com/mozilla-b2g/android-device-panda/blob/master/download-blobs.sh#L24

We need to fix the b2g build system to not rely on Google for tbpl builds.

Comment 4

[Aki Sasaki (not active)]

This is showing up because this link is now a 404 at Google:
wget -p -nd https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
--2013-02-05 10:54:46--  https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
Resolving dl.google.com... 74.125.226.206, 74.125.226.196, 74.125.226.199, ...
Connecting to dl.google.com|74.125.226.206|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-02-05 10:54:47 ERROR 404: Not Found.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

I sent a copy of the download-panda folder that contains the needed images.  This folder should be baked into the snapshot, or at the least deployed into the build root before running build.sh

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Lets keep this bug tracking patch-workaround to reopen trees as quickly as possible.


(In reply to Aki Sasaki [:aki] from comment #3)
> https://github.com/mozilla-b2g/android-device-panda/blob/master/download-
> blobs.sh#L24
> 
> We need to fix the b2g build system to not rely on Google for tbpl builds.

(In reply to Aki Sasaki [:aki] from comment #4)
> This is showing up because this link is now a 404 at Google:
> wget -p -nd
> https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
> --2013-02-05 10:54:46-- 
> https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
> Resolving dl.google.com... 74.125.226.206, 74.125.226.196, 74.125.226.199,
> ...
> Connecting to dl.google.com|74.125.226.206|:443... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2013-02-05 10:54:47 ERROR 404: Not Found.

Downloading files from external locations like this is not ok. 

Filed bug#838299 to track fixing b2g builds so that no external downloads are required, and situations like this do not impact production going forward.

Comment 7

[Aki Sasaki (not active)]

Landed https://hg.mozilla.org/integration/mozilla-inbound/rev/041e309db32e and am watching to see if it goes green.

If so, Ryan's going to merge to m-c and I'll land on b2g18 and b2g18_v1_0_0.

Comment 8

[Aki Sasaki (not active)]

http://hg.mozilla.org/releases/mozilla-b2g18/rev/28f8ec4fa439

Comment 9

[Aki Sasaki (not active)]

http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/e2c88d3108a5

We should go green at this point.
However, this is not the long term fix.  We should either:

* update the snapshot tooling to keep download-panda around for the next time around (I have a patch pending r?), or
* update the b2g_build.py script to download the download-panda tarball from tooltool and extract before starting the build.

Comment 10

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #9)
> http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/e2c88d3108a5
> 
> We should go green at this point.

Green on b2g18 and b2g18_v1_0_0

Comment 11

[Aki Sasaki (not active)]

Attachment: (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

http://hg.mozilla.org/integration/mozilla-inbound/rev/a69e0304a1f7

This probably needs some mozharness code to extract.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/041e309db32e

Comment 14

[Aki Sasaki (not active)]

Sorry, I should have [leave-open]ed for the comment 9 work.

Comment 15

[Aki Sasaki (not active)]

Attachment: (mozharness) additional_source_tarballs

I've never used tar without the appropriate 'z' or 'j' option (gz or bz2 respectively), but it looks like we do that elsewhere?

Comment 16

[Aki Sasaki (not active)]

Attachment: (configs) panda --additional-source-tarballs

I was going to create a releng-panda.py, then noticed that try had a separate config file.  Didn't want to also create a releng-try-panda.py, so command line options it is (til we get multiple config file support).

Comment 17

[Chris AtLee [:catlee]]

Comment on attachment 710456 [details] [diff] [review]
(mozharness) additional_source_tarballs

Review of attachment 710456 [details] [diff] [review]:
-----------------------------------------------------------------

we'll need to clobber all the panda builds (to refresh the .gonk_mtime file) for this to work

Comment 18

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/a69e0304a1f7

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 710456 [details] [diff] [review]
(mozharness) additional_source_tarballs

http://hg.mozilla.org/build/mozharness/rev/f0002e6fd811
Merged to production.

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 710457 [details] [diff] [review]
(configs) panda --additional-source-tarballs

http://hg.mozilla.org/build/buildbot-configs/rev/82d5d1a88f09

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 

Google 404 (see comment 4)

User impact if declined: 

The next panda snapshot builds will burn.

Testing completed: 

We were able to get the trees green by extracting this directly into the current snapshot.  We're just adding code to download/extract this specifically.

Risk to taking this patch (and alternatives if risky):

Low.

String or UUID changes made by this patch:

None.

Comment 22

[Aki Sasaki (not active)]

Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f

I'll get b2g18_v1_0_0 when things get a little greener on the two branches.

Comment 23

[Aki Sasaki (not active)]

Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f

I think we're just waiting for a reconfig at this point.

Comment 24

[Aki Sasaki (not active)]

Reconfig happened.  We should be good here.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

(In reply to Aki Sasaki [:aki] from comment #22)
> Comment on attachment 710385 [details] [diff] [review]
> (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt
> 
> https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f
> 
> I'll get b2g18_v1_0_0 when things get a little greener on the two branches.

For posterity:
https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/5cb954c79c62