All users were logged out of Bugzilla on October 13th, 2018

B2G panda builds need to include download-panda bits

RESOLVED FIXED in Firefox 21

Status

--
blocker
RESOLVED FIXED
6 years ago
5 months ago

People

(Reporter: RyanVM, Assigned: aki)

Tracking

other
B2G C4 (2jan on)
ARM
Gonk (Firefox OS)

Firefox Tracking Flags

(blocking-b2g:tef+, firefox19 unaffected, firefox20 unaffected, firefox21 fixed, b2g18 fixed, b2g18-v1.0.0 fixed, b2g18-v1.0.1 fixed)

Details

(Whiteboard: [treeclosure][leave-open][NPOTB])

Attachments

(3 attachments)

(Reporter)

Description

6 years ago
This is hitting us on all branches that build b2g. Originally this was thought to be related to the earlier scl3 issues, but we're still seeing failing builds after that was resolved.

https://tbpl.mozilla.org/php/getParsedLog.php?id=19459089&tree=Mozilla-Inbound

10:26:27     INFO -  pulling from http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central
10:26:27     INFO -  searching for changes
10:26:27     INFO -  adding changesets
10:26:27     INFO -  adding manifests
10:26:27     INFO -  adding file changes
10:26:27     INFO -  added 30 changesets with 109 changes to 96 files
10:26:27    ERROR -  abort: HTTP Error 500: Internal Server Error
10:26:27    ERROR -  Automation Error: hg not responding
10:26:27     INFO -  command: ERROR
10:26:27     INFO -  Traceback (most recent call last):
10:26:27     INFO -    File "<string>", line 40, in run_cmd
10:26:27     INFO -    File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
10:26:27     INFO -      raise CalledProcessError(retcode, cmd)
10:26:27     INFO -  CalledProcessError: Command '['hg', 'pull', 'http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central']' returned non-zero exit status 255
10:26:27     INFO -  command: END (15.56s elapsed)
10:26:27     INFO -  Problem pulling from mirror http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central
10:26:27     INFO -  Traceback (most recent call last):
10:26:27     INFO -    File "<string>", line 224, in pull
10:26:27     INFO -    File "<string>", line 246, in pull
10:26:27     INFO -    File "<string>", line 40, in run_cmd
10:26:27     INFO -    File "/usr/lib64/python2.6/subprocess.py", line 502, in check_call
10:26:27     INFO -      raise CalledProcessError(retcode, cmd)
10:26:27     INFO -  CalledProcessError: Command '['hg', 'pull', 'http://hg-internal.dmz.scl3.mozilla.com/integration/gaia-central']' returned non-zero exit status 255
10:26:27     INFO -  Pulling from mirrors failed; falling back to http://hg.mozilla.org/integration/gaia-central


and later


10:29:46     INFO -  Blob setup script has changed, re-running
10:29:46     INFO -  shasum: imgtec-panda-imm76i-67545da7.tgz: No such file or directory
10:29:46     INFO -  shasum: WARNING: 1 of 1 listed files could not be read
10:29:46     INFO -  imgtec-panda-imm76i-67545da7.tgz: FAILED open or read
10:29:46     INFO -    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
10:29:46     INFO -                                   Dload  Upload   Total   Spent    Left  Speed
10:29:47     INFO -  
10:29:47     INFO -   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
10:29:47     INFO - 108   869  108   869    0     0   1543      0 --:--:-- --:--:-- --:--:--  7427
10:29:47     INFO -  shasum: WARNING: 1 of 1 computed checksums did NOT match
10:29:47     INFO -  imgtec-panda-imm76i-67545da7.tgz: FAILED
10:29:47     INFO -  
10:29:47     INFO -  > Build failed! <
10:29:47     INFO -  Build with |./build.sh -j1| for better messages
10:29:47     INFO -  If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
10:29:47    ERROR - Return code: 1
10:29:47     INFO - Running command: ['mock_mozilla', '-r', u'mozilla-centos6-i386', '-q', '--cwd', '/builds/slave/b2g-m-in-panda-dep/build', '--unpriv', '--shell', u'/usr/bin/env "LESSOPEN=|/usr/bin/lesspipe.sh %s" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g-m-in-panda-dep/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130205102120 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g-m-in-panda-dep/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com PYTHONPATH=:/builds/slave/b2g-m-in-panda-dep/compare-locales/lib L10NBASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gecko-l10n "MOZ_CHROME_MULTILOCALE=es-ES pt-BR" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-625.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json MAIL=/var/spool/mail/cltbld ccache -s'] in /builds/slave/b2g-m-in-panda-dep/build
10:29:47     INFO - Copy/paste: mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g-m-in-panda-dep/build --unpriv --shell "/usr/bin/env \"LESSOPEN=|/usr/bin/lesspipe.sh %s\" LOGNAME=cltbld USER=cltbld PATH=/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin:/builds/slave/b2g-m-in-panda-dep/compare-locales/scripts SYMBOL_SERVER_USER=b2gbld CCACHE_UMASK=002 LANG=en_US.UTF-8 CCACHE_HASHDIR= TERM=linux SHELL=/bin/bash MOZ_BUILD_DATE=20130205102120 SHLVL=1 G_BROKEN_FILENAMES=1 HISTSIZE=1000 SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_b2g/ LOCALES_FILE=/builds/slave/b2g-m-in-panda-dep/build/gaia/locales/languages_dev.json SYMBOL_SERVER_HOST=symbols1.dmz.phx1.mozilla.com PYTHONPATH=:/builds/slave/b2g-m-in-panda-dep/compare-locales/lib L10NBASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gecko-l10n \"MOZ_CHROME_MULTILOCALE=es-ES pt-BR\" CCACHE_DIR=/builds/ccache CCACHE_COMPRESS=1 _=/tools/buildbot/bin/python CCACHE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build HOSTNAME=bld-linux64-ec2-625.build.aws-us-west-2.mozilla.com SYMBOL_SERVER_SSH_KEY=/home/mock_mozilla/.ssh/b2gbld_dsa LOCALE_BASEDIR=/builds/slave/b2g-m-in-panda-dep/build/gaia-l10n HISTCONTROL=ignoredups POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py PWD=/builds/slave/b2g-m-in-panda-dep PROPERTIES_FILE=/builds/slave/b2g-m-in-panda-dep/buildprops.json MAIL=/var/spool/mail/cltbld ccache -s"
10:29:47     INFO -  cache directory                     /builds/ccache
10:29:47     INFO -  cache hit (direct)                     0
10:29:47     INFO -  cache hit (preprocessed)               0
10:29:47     INFO -  cache miss                             0
10:29:47     INFO -  files in cache                    229449
10:29:47     INFO -  cache size                           9.3 Gbytes
10:29:47     INFO -  max cache size                      10.0 Gbytes
10:29:48     INFO - Return code: 0
10:29:48    FATAL - failed to build
10:29:48    FATAL - Exiting 2
(Reporter)

Comment 1

6 years ago
Forgot to note that m-c, inbound, b2g18, and b2g18_v1_0_0 are all closed until this is resolved.
found in triage.
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: catlee
Whiteboard: [treeclosure]
(Assignee)

Comment 3

6 years ago
https://github.com/mozilla-b2g/android-device-panda/blob/master/download-blobs.sh#L24

We need to fix the b2g build system to not rely on Google for tbpl builds.
(Assignee)

Comment 4

6 years ago
This is showing up because this link is now a 404 at Google:
wget -p -nd https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
--2013-02-05 10:54:46--  https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
Resolving dl.google.com... 74.125.226.206, 74.125.226.196, 74.125.226.199, ...
Connecting to dl.google.com|74.125.226.206|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-02-05 10:54:47 ERROR 404: Not Found.
I sent a copy of the download-panda folder that contains the needed images.  This folder should be baked into the snapshot, or at the least deployed into the build root before running build.sh
Lets keep this bug tracking patch-workaround to reopen trees as quickly as possible.


(In reply to Aki Sasaki [:aki] from comment #3)
> https://github.com/mozilla-b2g/android-device-panda/blob/master/download-
> blobs.sh#L24
> 
> We need to fix the b2g build system to not rely on Google for tbpl builds.

(In reply to Aki Sasaki [:aki] from comment #4)
> This is showing up because this link is now a 404 at Google:
> wget -p -nd
> https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
> --2013-02-05 10:54:46-- 
> https://dl.google.com/dl/android/aosp/imgtec-panda-imm76i-67545da7.tgz
> Resolving dl.google.com... 74.125.226.206, 74.125.226.196, 74.125.226.199,
> ...
> Connecting to dl.google.com|74.125.226.206|:443... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2013-02-05 10:54:47 ERROR 404: Not Found.

Downloading files from external locations like this is not ok. 

Filed bug#838299 to track fixing b2g builds so that no external downloads are required, and situations like this do not impact production going forward.
Blocks: 838299
(Assignee)

Comment 7

6 years ago
Landed https://hg.mozilla.org/integration/mozilla-inbound/rev/041e309db32e and am watching to see if it goes green.

If so, Ryan's going to merge to m-c and I'll land on b2g18 and b2g18_v1_0_0.
(Assignee)

Comment 9

6 years ago
http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/e2c88d3108a5

We should go green at this point.
However, this is not the long term fix.  We should either:

* update the snapshot tooling to keep download-panda around for the next time around (I have a patch pending r?), or
* update the b2g_build.py script to download the download-panda tarball from tooltool and extract before starting the build.
(Reporter)

Updated

6 years ago
status-b2g18: --- → fixed
status-b2g18-v1.0.0: --- → fixed
status-firefox19: --- → unaffected
status-firefox20: --- → unaffected
status-firefox21: --- → fixed
Target Milestone: --- → B2G C4 (2jan on)
(Assignee)

Updated

6 years ago
Summary: B2G panda builds failing with Internal Server Errors → B2G panda builds need to include download-panda bits
(Assignee)

Comment 10

6 years ago
(In reply to Aki Sasaki [:aki] from comment #9)
> http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/e2c88d3108a5
> 
> We should go green at this point.

Green on b2g18 and b2g18_v1_0_0
(Assignee)

Comment 11

6 years ago
Created attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt
Assignee: nobody → aki
Attachment #710385 - Flags: review?(catlee)

Updated

6 years ago
Attachment #710385 - Flags: review?(catlee) → review+
(Assignee)

Comment 12

6 years ago
Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

http://hg.mozilla.org/integration/mozilla-inbound/rev/a69e0304a1f7

This probably needs some mozharness code to extract.
Attachment #710385 - Flags: checked-in+
(Reporter)

Comment 13

6 years ago
https://hg.mozilla.org/mozilla-central/rev/041e309db32e
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Assignee)

Comment 14

6 years ago
Sorry, I should have [leave-open]ed for the comment 9 work.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 15

6 years ago
Created attachment 710456 [details] [diff] [review]
(mozharness) additional_source_tarballs

I've never used tar without the appropriate 'z' or 'j' option (gz or bz2 respectively), but it looks like we do that elsewhere?
Attachment #710456 - Flags: review?(catlee)
(Assignee)

Comment 16

6 years ago
Created attachment 710457 [details] [diff] [review]
(configs) panda --additional-source-tarballs

I was going to create a releng-panda.py, then noticed that try had a separate config file.  Didn't want to also create a releng-try-panda.py, so command line options it is (til we get multiple config file support).
Attachment #710457 - Flags: review?(catlee)
Comment on attachment 710456 [details] [diff] [review]
(mozharness) additional_source_tarballs

Review of attachment 710456 [details] [diff] [review]:
-----------------------------------------------------------------

we'll need to clobber all the panda builds (to refresh the .gonk_mtime file) for this to work
Attachment #710456 - Flags: review?(catlee) → review+

Updated

6 years ago
Attachment #710457 - Flags: review?(catlee) → review+
https://hg.mozilla.org/mozilla-central/rev/a69e0304a1f7
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
(Assignee)

Updated

6 years ago
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [treeclosure] → [treeclosure][leave-open]
(Assignee)

Comment 19

6 years ago
Comment on attachment 710456 [details] [diff] [review]
(mozharness) additional_source_tarballs

http://hg.mozilla.org/build/mozharness/rev/f0002e6fd811
Merged to production.
Attachment #710456 - Flags: checked-in+
(Assignee)

Comment 20

6 years ago
Comment on attachment 710457 [details] [diff] [review]
(configs) panda --additional-source-tarballs

http://hg.mozilla.org/build/buildbot-configs/rev/82d5d1a88f09
Attachment #710457 - Flags: checked-in+
(Assignee)

Comment 21

6 years ago
Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 

Google 404 (see comment 4)

User impact if declined: 

The next panda snapshot builds will burn.

Testing completed: 

We were able to get the trees green by extracting this directly into the current snapshot.  We're just adding code to download/extract this specifically.

Risk to taking this patch (and alternatives if risky):

Low.

String or UUID changes made by this patch:

None.
Attachment #710385 - Flags: approval-mozilla-b2g18?

Updated

6 years ago
Attachment #710385 - Flags: approval-mozilla-b2g18? → approval-mozilla-b2g18+

Updated

6 years ago
blocking-b2g: --- → tef+
Whiteboard: [treeclosure][leave-open] → [treeclosure][leave-open][NPOTB]
(Assignee)

Comment 22

6 years ago
Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f

I'll get b2g18_v1_0_0 when things get a little greener on the two branches.
(Assignee)

Comment 23

6 years ago
Comment on attachment 710385 [details] [diff] [review]
(inbound) add download-panda.tar.bz2 to releng-pandaboard.tt

https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f

I think we're just waiting for a reconfig at this point.
(Assignee)

Comment 24

6 years ago
Reconfig happened.  We should be good here.
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 25

6 years ago
(In reply to Aki Sasaki [:aki] from comment #22)
> Comment on attachment 710385 [details] [diff] [review]
> (inbound) add download-panda.tar.bz2 to releng-pandaboard.tt
> 
> https://hg.mozilla.org/releases/mozilla-b2g18/rev/bc294cb0454f
> 
> I'll get b2g18_v1_0_0 when things get a little greener on the two branches.

For posterity:
https://hg.mozilla.org/releases/mozilla-b2g18_v1_0_0/rev/5cb954c79c62
(Reporter)

Updated

6 years ago
status-b2g18-v1.0.1: --- → fixed
Product: mozilla.org → Release Engineering
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.