663108 - Try Server reports wrong .zip file as downloadable build

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[David Humphrey (:humph)]

In bug 543463 I'm adding Makefile changes to have a jsshell.zip get packaged and uploaded with our builds.  My try server builds are coming back with this message on every platform:

"It should be available for download at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/jsshell.zip"

While it's helpful to me to get the link to the jsshell.zip, it's unlikely to be what 99% of devs expect when they submit a firefox build.

Comment 1

[Lukas Blakk [:lsblakk] use ?needinfo]

http://production-master02.build.mozilla.org:8011/builders/Linux%20try%20build/builds/709

shows that packageUrl is being set to the jsshell.zip:

packageUrl 	http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/jsshell.zip

packageUrl is what the try_mailer uses http://hg.mozilla.org/build/buildbotcustom/file/89bda47f09ef/bin/try_mailer.py#l99 to decide to send that particular message and download url

this is happening in the scratchbox-setproperty step:

(view as text)

make upload
 in dir /builds/slave/try-lnx/build/obj-firefox (timeout 2400 secs)
 watching logfiles {}
 argv: ['make', 'upload']
 environment:
  CC=/tools/gcc/bin/gcc
  CCACHE_BASEDIR=/builds/slave/try-lnx
  CCACHE_COMPRESS=1
  CCACHE_DIR=/builds/ccache
  CCACHE_UMASK=002
  CVS_RSH=ssh
  CXX=/tools/gcc/bin/g++
  DISPLAY=:2
  G_BROKEN_FILENAMES=1
  HG_SHARE_BASE_DIR=/builds/hg-shared
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=try-linux-slave11
  INPUTRC=/etc/inputrc
  JAVA_HOME=/builds/jdk
  LANG=en_US.UTF-8
  LC_ALL=C
  LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
  MAIL=/var/spool/mail/cltbld
  MOZ_CRASHREPORTER_NO_REPORT=1
  MOZ_OBJDIR=obj-firefox
  PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
  POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py
  POST_UPLOAD_CMD=post_upload.py --tinderbox-builds-dir [e-mail]-9f4e762e92f0 -p firefox -i 20110608190046 --revision 9f4e762e92f0 --who [e-mail] --builddir try-linux --release-to-try-builds
  PWD=/builds/slave/try-lnx/build/obj-firefox
  SHELL=/bin/bash
  SHLVL=1
  SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
  SYMBOL_SERVER_HOST=dm-symbolpush01.mozilla.org
  SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/
  SYMBOL_SERVER_SSH_KEY=/home/cltbld/.ssh/ffxbld_dsa
  SYMBOL_SERVER_USER=ffxbld
  TBOX_CLIENT_CVS_DIR=/builds/tinderbox/mozilla/tools
  TERM=linux
  TINDERBOX_OUTPUT=1
  UPLOAD_HOST=stage.mozilla.org
  UPLOAD_SSH_KEY=~/.ssh/trybld_dsa
  UPLOAD_TO_TEMP=1
  UPLOAD_USER=trybld
  USER=cltbld
  _=/tools/python/bin/python
 using PTY: False
make[1]: Entering directory `/builds/slave/try-lnx/build/obj-firefox/browser/installer'
CHECKSUM FILE START
2213a2b2cd90964623cec2cd7974020758c83e9922801fa92d5432ccdc5e3d878412adb20a17d14a384a37887387cb33d93e2556f67c5dd24d8dab6a157e9e74 sha512 15156776 firefox-7.0a1.en-US.linux-i686.tar.bz2
33102f68f85b74afc6841f1e0f141c869ca00c9f62777626c067e0d0098ece08eba63288797ed87bf66553075c79a2e05dec811507622fed944b2688f4a68a30 sha512 217211 linux-i686/xpi/firefox-7.0a1.en-US.langpack.xpi
3b6e70e11f73c123fb3f17603215adac97a57a64ed8d0ccb91b4953d940aaaa00d3a1758b6b8d4e06b588f1cb661680163126ac3b28a4d029b2639cdc5ef0cde sha512 48433079 firefox-7.0a1.en-US.linux-i686.tests.zip
6eec148b92f9ab1afbd7dee66ca18fb26da25907987ad57f676d82e956a4b201e644eb63ff7b3288862adc8f2ff1067b3f7e6c64da9d88461c0e604a473e15b3 sha512 20974565 firefox-7.0a1.en-US.linux-i686.crashreporter-symbols.zip
31a0395344da5b3a06f41ea274efb613ab15dae048fc19b3ce616ff3e1c082662d5f4a4e7cf6ff39ad9ab0a30aac81e22f86e51a3ec8c40d69a67624661eb6d4 sha512 58 firefox-7.0a1.en-US.linux-i686.txt
943661d15fea807dcf86c2fe87c9de8268b4774cd71143e4f2e474343486d344faa864bcc2a6efd9935146cbdb154ac64af480b56ed7b5147790248524b59f7d sha512 11156128 jsshell.zip
CHECKSUM FILE END
/tools/python/bin/python2.5 /builds/slave/try-lnx/build/build/upload.py --base-path ../../dist \
		"../../dist/firefox-7.0a1.en-US.linux-i686.tar.bz2"   "../../dist/linux-i686/xpi/firefox-7.0a1.en-US.langpack.xpi"  "../../dist/firefox-7.0a1.en-US.linux-i686.tests.zip" "../../dist/firefox-7.0a1.en-US.linux-i686.crashreporter-symbols.zip"  "../../dist//firefox-7.0a1.en-US.linux-i686.txt" "../../dist/jsshell.zip"  \
		"../../dist//firefox-7.0a1.en-US.linux-i686.checksums"
http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/firefox-7.0a1.en-US.linux-i686.tar.bz2
http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/firefox-7.0a1.en-US.langpack.xpi
http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/firefox-7.0a1.en-US.linux-i686.tests.zip
http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/firefox-7.0a1.en-US.linux-i686.crashreporter-symbols.zip
http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/firefox-7.0a1.en-US.linux-i686.txt
http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/jsshell.zip
http://stage.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-9f4e762e92f0/try-linux/firefox-7.0a1.en-US.linux-i686.checksums
Uploading /builds/slave/try-lnx/build/obj-firefox/dist/firefox-7.0a1.en-US.linux-i686.tar.bz2
Uploading /builds/slave/try-lnx/build/obj-firefox/dist/linux-i686/xpi/firefox-7.0a1.en-US.langpack.xpi
Uploading /builds/slave/try-lnx/build/obj-firefox/dist/firefox-7.0a1.en-US.linux-i686.tests.zip
Uploading /builds/slave/try-lnx/build/obj-firefox/dist/firefox-7.0a1.en-US.linux-i686.crashreporter-symbols.zip
Uploading /builds/slave/try-lnx/build/obj-firefox/dist/firefox-7.0a1.en-US.linux-i686.txt
Uploading /builds/slave/try-lnx/build/obj-firefox/dist/jsshell.zip
Uploading /builds/slave/try-lnx/build/obj-firefox/dist/firefox-7.0a1.en-US.linux-i686.checksums
Running post-upload command: post_upload.py --tinderbox-builds-dir [e-mail]-9f4e762e92f0 -p firefox -i 20110608190046 --revision 9f4e762e92f0 --who [e-mail] --builddir try-linux --release-to-try-builds
Upload complete
make[1]: Leaving directory `/builds/slave/try-lnx/build/obj-firefox/browser/installer'
program finished with exit code 0
elapsedTime=31.164266

Comment 2

[Lukas Blakk [:lsblakk] use ?needinfo]

looks like this is related to the mobile merge work by jhford, throwing it to him to take a closer look.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #2)
> looks like this is related to the mobile merge work by jhford, throwing it
> to him to take a closer look.

scratchbox-property is just a wrapper we introduced in 557260 to optionally abstract scratchbox.  There is no functional difference between ScratchboxProperty and SetProperty.

I have tested that function and it is indeed picking out the jsshell.zip file with the stdio from that make upload step.  Not sure why, but I'll look at it on the train ride in.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

I don't have my dev environment ready (i just woke up and need to shower for train ride in) but I have fixed the code.  The problem is that any file that is not the symbols, tests or unsigned apk file is set to the packageUrl.  Without sending in some sort of pattern to the parse_make_upload function or hardcoding search strings, we need to account for all non main package files that will be in the list of files to upload.  I will prepare a patch for the function below, but this is the body of what buildbotcustom.process.factory.parse_make_upload needs to be to fix this problem.

This is also unrelated to the mobile merge or scratchbox-setproperty.

A more (though possible less) robust solution could be to take the common prefix string from the symbols and tests archive and use that to search for the main package.

def parse_make_upload(rc, stdout, stderr):
    ''' This function takes the output and return code from running
    the upload make target and returns a dictionary of important
    file urls.'''
    retval = {}
    for m in re.findall("^(http://.*?\.(?:tar\.bz2|dmg|zip|apk))",
                        "\n".join([stdout, stderr]), re.M):
        if m.endswith("crashreporter-symbols.zip"):
            retval['symbolsUrl'] = m
        elif m.endswith("tests.tar.bz2") or m.endswith("tests.zip"):
            retval['testsUrl'] = m
        elif m.endswith("apk") and 'unsigned' in m:
            retval['unsignedApkUrl'] = m
        elif m.endswith("jsshell.zip"):
            retval['jsshellUrl'] = m
        else:
            retval['packageUrl'] = m
    return retval

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbotcustom v1

this patch teaches parse_make_upload about the jsshell.zip files

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 538265 [details] [diff] [review]
buildbotcustom v1

http://hg.mozilla.org/build/buildbotcustom/rev/7f10a3e2e602

Comment 7

[Aki Sasaki (not active)]

This made it to production today.

Comment 8

[David Humphrey (:humph)]

FWIW, and possibly just a timing issue with build-start vs -end, but as of 6 minutes ago, I got:

It should be available for download at 
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-476a0609ae75/try-win32/jsshell.zip

Comment 9

[Aki Sasaki (not active)]

The code will only work for builds started after the reconfig; do you know when it started?

Comment 10

[David Humphrey (:humph)]

Yeah, I figured.  I started this one: Thu Jun 9 10:50:55 2011 PDT

http://tbpl.mozilla.org/?tree=Try&rev=476a0609ae75

Comment 11

[Aki Sasaki (not active)]

That's before I started reconfiging, so this patch wasn't live yet.
Hopefully the next one will work.

Comment 12

[Joachim Herb]

Is it possible that this patch breaks the try server mozmill and xpcshell tests for Thunderbird?

The last build (for me) which worked was:
http://build.mozillamessaging.com/tinderboxpushlog/?tree=ThunderbirdTry&rev=7a2525c8fede

The suddenly the mozmill and xpcshell tests stopped to work, e.g.
http://tinderbox.mozilla.org/showlog.cgi?log=ThunderbirdTry/1309108562.1309108626.20609.gz&fulltext=1

The problem seems to be that a file jsshell-linux-i686.crashreporter-symbols.zip is tried to be downloaded from the build server, which does not exist. In the older build that file was generated.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

The MoMo try server will need the patch from this bug too.