Closed
Bug 838261
Opened 11 years ago
Closed 11 years ago
B2G panda builds need to include download-panda bits
Categories
(Release Engineering :: General, defect)
Tracking
(blocking-b2g:tef+, firefox19 unaffected, firefox20 unaffected, firefox21 fixed, b2g18 fixed, b2g18-v1.0.0 fixed, b2g18-v1.0.1 fixed)
Tracking | Status | |
---|---|---|
firefox19 | --- | unaffected |
firefox20 | --- | unaffected |
firefox21 | --- | fixed |
b2g18 | --- | fixed |
b2g18-v1.0.0 | --- | fixed |
b2g18-v1.0.1 | --- | fixed |
People
(Reporter: RyanVM, Assigned: mozilla)
References
Details
(Whiteboard: [treeclosure][leave-open][NPOTB])
Attachments
(3 files)
652 bytes,
patch
|
catlee
:
review+
akeybl
:
approval-mozilla-b2g18+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
4.36 KB,
patch
|
catlee
:
review+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
2.75 KB,
patch
|
catlee
:
review+
mozilla
:
checked-in+
|
Details | Diff | Splinter Review |
This is hitting us on all branches that build b2g. Originally this was thought to be related to the earlier scl3 issues, but we're still seeing failing builds after that was resolved. https://tbpl.mozilla.org/php/getParsedLog.php?id=19459089&tree=Mozilla-Inbound 10:26:27 INFO - pulling from http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central 10:26:27 INFO - searching for changes 10:26:27 INFO - adding changesets 10:26:27 INFO - adding manifests 10:26:27 INFO - adding file changes 10:26:27 INFO - added 30 changesets with 109 changes to 96 files 10:26:27 ERROR - abort: HTTP Error 500: Internal Server Error 10:26:27 ERROR - Automation Error: hg not responding 10:26:27 INFO - command: ERROR 10:26:27 INFO - Traceback (most recent call last): 10:26:27 INFO - File "<string>", line 40, in run_cmd 10:26:27 INFO - File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call 10:26:27 INFO - raise CalledProcessError(retcode, cmd) 10:26:27 INFO - CalledProcessError: Command '['hg', 'pull', 'http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central']' returned non-zero exit status 255 10:26:27 INFO - command: END (15.56s elapsed) 10:26:27 INFO - Problem pulling from mirror http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central 10:26:27 INFO - Traceback (most recent call last): 10:26:27 INFO - File "<string>", line 224, in pull 10:26:27 INFO - File "<string>", line 246, in pull 10:26:27 INFO - File "<string>", line 40, in run_cmd 10:26:27 INFO - File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call 10:26:27 INFO - raise CalledProcessError(retcode, cmd) 10:26:27 INFO - CalledProcessError: Command '['hg', 'pull', 'http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central']' returned non-zero exit status 255 10:26:27 INFO - Pulling from mirrors failed; falling back to http://hg.mozilla.org/integration/gaia-central and later 10:29:46 INFO - Blob setup script has changed, re-running 10:29:46 INFO - shasum: imgtec-panda-imm76i-67545da7.tgz: No such file or directory 10:29:46 INFO - shasum: WARNING: 1 of 1 listed files could not be read 10:29:46 INFO - imgtec-panda-imm76i-67545da7.tgz: FAILED open or read 10:29:46 INFO - % Total % Received % Xferd Average Speed Time Time Time Current 10:29:46 INFO - Dload Upload Total Spent Left Speed 10:29:47 INFO - 10:29:47 INFO - 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 10:29:47 INFO - 108 869 108 869 0 0 1543 0 --:--:-- --:--:-- --:--:-- 7427 10:29:47 INFO - shasum: WARNING: 1 of 1 computed checksums did NOT match 10:29:47 INFO - imgtec-panda-imm76i-67545da7.tgz: FAILED 10:29:47 INFO - 10:29:47 INFO - > Build failed! < 10:29:47 INFO - Build with |./build.sh -j1| for better messages 10:29:47 INFO - If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else. 10:29:47 ERROR - Return code: 1 10:29:47 INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g-m-in-panda-dep/build', '--unpriv', '--shell', u'/usr/bin/env "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g-m-in-panda-dep/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130205102120 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g-m-in-panda-dep/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com PYTHONPATH=:/builds/slave/b2g-m-in-panda-dep/compare-locales/lib L10NBASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gecko-l10n "MOZ_CHROME_MULTILOCALE=es-ES pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-625.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json MAIL=/var/spool/mail/cltbld ccache -s'] in /builds/slave/b2g-m-in-panda-dep/build 10:29:47 INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g-m-in-panda-dep/build --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g-m-in-panda-dep/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130205102120 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g-m-in-panda-dep/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com PYTHONPATH=:/builds/slave/b2g-m-in-panda-dep/compare-locales/lib L10NBASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gecko-l10n \"MOZ_CHROME_MULTILOCALE=es-ES pt-BR\" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-625.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json MAIL=/var/spool/mail/cltbld ccache -s" 10:29:47 INFO - cache directory /builds/ccache 10:29:47 INFO - cache hit (direct) 0 10:29:47 INFO - cache hit (preprocessed) 0 10:29:47 INFO - cache miss 0 10:29:47 INFO - files in cache 229449 10:29:47 INFO - cache size 9.3 Gbytes 10:29:47 INFO - max cache size 10.0 Gbytes 10:29:48 INFO - Return code: 0 10:29:48 FATAL - failed to build 10:29:48 FATAL - Exiting 2
Reporter | ||
Comment 1•11 years ago
|
||
Forgot to note that m-c, inbound, b2g18, and b2g18_v1_0_0 are all closed until this is resolved.
Comment 2•11 years ago
|
||
found in triage.
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
Whiteboard: [treeclosure]
Assignee | ||
Comment 3•11 years ago
|
||
https://github.com/mozilla-b2g/android-device-panda/blob/master/download-blobs.sh#L24 We need to fix the b2g build system to not rely on Google for tbpl builds.
Assignee | ||
Comment 4•11 years ago
|
||
This is showing up because this link is now a 404 at Google: wget -p -nd https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz --2013-02-05 10:54:46-- https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz Resolving dl.google.com... 74.125.226.206, 74.125.226.196, 74.125.226.199, ... Connecting to dl.google.com|74.125.226.206|:443... connected. HTTP request sent, awaiting response... 404 Not Found 2013-02-05 10:54:47 ERROR 404: Not Found.
Comment 5•11 years ago
|
||
I sent a copy of the download-panda folder that contains the needed images. This folder should be baked into the snapshot, or at the least deployed into the build root before running build.sh
Comment 6•11 years ago
|
||
Lets keep this bug tracking patch-workaround to reopen trees as quickly as possible. (In reply to Aki Sasaki [:aki] from comment #3) > https://github.com/mozilla-b2g/android-device-panda/blob/master/download- > blobs.sh#L24 > > We need to fix the b2g build system to not rely on Google for tbpl builds. (In reply to Aki Sasaki [:aki] from comment #4) > This is showing up because this link is now a 404 at Google: > wget -p -nd > https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz > --2013-02-05 10:54:46-- > https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz > Resolving dl.google.com... 74.125.226.206, 74.125.226.196, 74.125.226.199, > ... > Connecting to dl.google.com|74.125.226.206|:443... connected. > HTTP request sent, awaiting response... 404 Not Found > 2013-02-05 10:54:47 ERROR 404: Not Found. Downloading files from external locations like this is not ok. Filed bug#838299 to track fixing b2g builds so that no external downloads are required, and situations like this do not impact production going forward.
Blocks: 838299
Assignee | ||
Comment 7•11 years ago
|
||
Landed https://hg.mozilla.org/integration/mozilla-inbound/rev/041e309db32e and am watching to see if it goes green. If so, Ryan's going to merge to m-c and I'll land on b2g18 and b2g18_v1_0_0.
Assignee | ||
Comment 8•11 years ago
|
||
http://hg.mozilla.org/releases/mozilla-b2g18/rev/28f8ec4fa439
Assignee | ||
Comment 9•11 years ago
|
||
http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/e2c88d3108a5 We should go green at this point. However, this is not the long term fix. We should either: * update the snapshot tooling to keep download-panda around for the next time around (I have a patch pending r?), or * update the b2g_build.py script to download the download-panda tarball from tooltool and extract before starting the build.
Reporter | ||
Updated•11 years ago
|
status-b2g18:
--- → fixed
status-b2g18-v1.0.0:
--- → fixed
status-firefox19:
--- → unaffected
status-firefox20:
--- → unaffected
status-firefox21:
--- → fixed
Target Milestone: --- → B2G C4 (2jan on)
Assignee | ||
Updated•11 years ago
|
Summary: B2G panda builds failing with Internal Server Errors → B2G panda builds need to include download-panda bits
Assignee | ||
Comment 10•11 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #9) > http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/e2c88d3108a5 > > We should go green at this point. Green on b2g18 and b2g18_v1_0_0
Assignee | ||
Comment 11•11 years ago
|
||
Assignee: nobody → aki
Attachment #710385 -
Flags: review?(catlee)
Updated•11 years ago
|
Attachment #710385 -
Flags: review?(catlee) → review+
Assignee | ||
Comment 12•11 years ago
|
||
Comment on attachment 710385 [details] [diff] [review] (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt http://hg.mozilla.org/integration/mozilla-inbound/rev/a69e0304a1f7 This probably needs some mozharness code to extract.
Attachment #710385 -
Flags: checked-in+
Reporter | ||
Comment 13•11 years ago
|
||
https://hg.mozilla.org/mozilla-central/rev/041e309db32e
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 14•11 years ago
|
||
Sorry, I should have [leave-open]ed for the comment 9 work.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 15•11 years ago
|
||
I've never used tar without the appropriate 'z' or 'j' option (gz or bz2 respectively), but it looks like we do that elsewhere?
Attachment #710456 -
Flags: review?(catlee)
Assignee | ||
Comment 16•11 years ago
|
||
I was going to create a releng-panda.py, then noticed that try had a separate config file. Didn't want to also create a releng-try-panda.py, so command line options it is (til we get multiple config file support).
Attachment #710457 -
Flags: review?(catlee)
Comment 17•11 years ago
|
||
Comment on attachment 710456 [details] [diff] [review] (mozharness) additional_source_tarballs Review of attachment 710456 [details] [diff] [review]: ----------------------------------------------------------------- we'll need to clobber all the panda builds (to refresh the .gonk_mtime file) for this to work
Attachment #710456 -
Flags: review?(catlee) → review+
Updated•11 years ago
|
Attachment #710457 -
Flags: review?(catlee) → review+
Comment 18•11 years ago
|
||
https://hg.mozilla.org/mozilla-central/rev/a69e0304a1f7
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Assignee | ||
Updated•11 years ago
|
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [treeclosure] → [treeclosure][leave-open]
Assignee | ||
Comment 19•11 years ago
|
||
Comment on attachment 710456 [details] [diff] [review] (mozharness) additional_source_tarballs http://hg.mozilla.org/build/mozharness/rev/f0002e6fd811 Merged to production.
Attachment #710456 -
Flags: checked-in+
Assignee | ||
Comment 20•11 years ago
|
||
Comment on attachment 710457 [details] [diff] [review] (configs) panda --additional-source-tarballs http://hg.mozilla.org/build/buildbot-configs/rev/82d5d1a88f09
Attachment #710457 -
Flags: checked-in+
Assignee | ||
Comment 21•11 years ago
|
||
Comment on attachment 710385 [details] [diff] [review] (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt [Approval Request Comment] Bug caused by (feature/regressing bug #): Google 404 (see comment 4) User impact if declined: The next panda snapshot builds will burn. Testing completed: We were able to get the trees green by extracting this directly into the current snapshot. We're just adding code to download/extract this specifically. Risk to taking this patch (and alternatives if risky): Low. String or UUID changes made by this patch: None.
Attachment #710385 -
Flags: approval-mozilla-b2g18?
Updated•11 years ago
|
Attachment #710385 -
Flags: approval-mozilla-b2g18? → approval-mozilla-b2g18+
Updated•11 years ago
|
blocking-b2g: --- → tef+
Whiteboard: [treeclosure][leave-open] → [treeclosure][leave-open][NPOTB]
Assignee | ||
Comment 22•11 years ago
|
||
Comment on attachment 710385 [details] [diff] [review] (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f I'll get b2g18_v1_0_0 when things get a little greener on the two branches.
Assignee | ||
Comment 23•11 years ago
|
||
Comment on attachment 710385 [details] [diff] [review] (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f I think we're just waiting for a reconfig at this point.
Assignee | ||
Comment 24•11 years ago
|
||
Reconfig happened. We should be good here.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 25•11 years ago
|
||
(In reply to Aki Sasaki [:aki] from comment #22) > Comment on attachment 710385 [details] [diff] [review] > (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt > > https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f > > I'll get b2g18_v1_0_0 when things get a little greener on the two branches. For posterity: https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/5cb954c79c62
Reporter | ||
Updated•11 years ago
|
status-b2g18-v1.0.1:
--- → fixed
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•