510042 - l10n repacks on change do not upload

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Armen [:armenzg]]

Since we turned the l10n nightly updates the repackages on change for "fr" and "pt-PT" failed to upload.

We might want to fix this together with bug 509937.

I will need commit access for x-testing or someone to help me with it.

make l10n-upload-pt-PT
 in dir /builds/moz2_slave/mozilla-central-linux-l10n-dep/build/mozilla-central/browser/locales (timeout 1200 secs)
 watching logfiles {}
 argv: ['make', 'l10n-upload-pt-PT']
 environment:
  AB_CD=pt-PT
  CC=/tools/gcc/bin/gcc
  CVS_RSH=ssh
  CXX=/tools/gcc/bin/g++
  G_BROKEN_FILENAMES=1
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=moz2-linux-slave02.build.mozilla.org
  INPUTRC=/etc/inputrc
  JAVA_HOME=/builds/jdk
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
  MAIL=/var/spool/mail/cltbld
  PATH=/opt/local/bin:/tools/buildbot/bin:/tools/twisted/bin:/tools/twisted-core/bin:/tools/python/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
  POST_UPLOAD_CMD=post_upload.py -p firefox  -b mozilla-central-l10n  -i 20090812031201 --release-to-tinderbox-dated-builds
  PWD=/builds/moz2_slave/mozilla-central-linux-l10n-dep/build/mozilla-central/browser/locales
  PYTHONHOME=/tools/python
  PYTHONPATH=/tools/buildbotcustom:/tools/buildbot/lib/python2.5/site-packages:/tools/twisted/lib/python2.5/site-packages:/tools/twisted-core/lib/python2.5/site-packages:/tools/zope-interface/lib/python2.5/site-packages/
  SHELL=/bin/bash
  SHLVL=1
  SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
  TBOX_CLIENT_CVS_DIR=/builds/tinderbox/mozilla/tools
  TERM=linux
  UPLOAD_HOST=stage.mozilla.org
  UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa
  UPLOAD_TO_TEMP=1
  UPLOAD_USER=ffxbld
  USER=cltbld
  _=/tools/buildbot/bin/buildbot
 closing stdin
 using PTY: True
/tools/python/bin/python2.5 ../../build/upload.py --base-path ../../dist "../../dist/firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2" ../../dist/install/firefox-3.6a2pre.pt-PT.langpack.xpi
Uploading /builds/moz2_slave/mozilla-central-linux-l10n-dep/build/mozilla-central/dist/firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2

firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2       0%    0     0.0KB/s   --:-- ETA
firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2     100% 9748KB   9.5MB/s   00:00    
Uploading /builds/moz2_slave/mozilla-central-linux-l10n-dep/build/mozilla-central/dist/install/firefox-3.6a2pre.pt-PT.langpack.xpi

firefox-3.6a2pre.pt-PT.langpack.xpi             0%    0     0.0KB/s   --:-- ETA
firefox-3.6a2pre.pt-PT.langpack.xpi           100%  121KB 121.3KB/s   00:00    
Running post-upload command: post_upload.py -p firefox  -b mozilla-central-l10n  -i 20090812031201 --release-to-tinderbox-dated-builds
Encountered error while uploading
Command ['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_dsa', '[e-mail]', 'post_upload.py -p firefox  -b mozilla-central-l10n  -i 20090812031201 --release-to-tinderbox-dated-builds "/tmp/tmp.UtXHOI1211/" "/tmp/tmp.UtXHOI1211//firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2" "/tmp/tmp.UtXHOI1211/install/firefox-3.6a2pre.pt-PT.langpack.xpi"'] returned non-zero exit code: 1
make: *** [l10n-upload-pt-PT] Error 2
program finished with exit code 2
elapsedTime=1.863500

Comment 1

[Axel Hecht [:Pike]]

/tmp/tmp.UtXHOI1211//firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2
                   ^^

would be my guess, can you compare that path on a working build? Double slash usually indicates a problem with a missing path.

Comment 2

[Armen [:armenzg]]

(In reply to comment #1)
> /tmp/tmp.UtXHOI1211//firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2
>                    ^^
> 
> would be my guess, can you compare that path on a working build? Double slash
> usually indicates a problem with a missing path.

Thanks for the pointer I will look into that.

I just wanted to note that this problem has been introduced by turning the l10n nightly updates on. Even though we tested the repackage on change scenario this testing was done long ago and things might have changed slightly.
I believe it is better to fix this rather than pulling back the l10n nightly updates.

I will work on this either tomorrow or Friday

Comment 3

[Armen [:armenzg]]

(In reply to comment #1)
> /tmp/tmp.UtXHOI1211//firefox-3.6a2pre.pt-PT.linux-i686.tar.bz2
>                    ^^
> 
> would be my guess, can you compare that path on a working build? Double slash
> usually indicates a problem with a missing path.
I tried to compare it with a working one but that line only shows up when there is a problem:
|  Running post-upload command: post_upload.py -p firefox  -b mozilla-central-l10n  -i 20090812031634 --release-to-tinderbox-dated-builds
|  Encountered error while uploading
|  Command ['ssh', '-o', 'IdentityFile=~/.ssh/ffxbld_dsa', '[e-mail]', 'post_upload.py -p firefox  -b mozilla-central-l10n  -i 20090812031634 --release-to-tinderbox-dated-builds "/tmp/tmp.INJzZl7353/" "/tmp/tmp.INJzZl7353//firefox-3.6a2pre.ru.mac.dmg" "/tmp/tmp.INJzZl7353/install/firefox-3.6a2pre.ru.langpack.xpi"'] returned non-zero exit code: 1
|  make: *** [l10n-upload-ru] Error 2

Comment 4

[Armen [:armenzg]]

Attachment: fixes the post_upload command used by repacks on change

This patch will make all repacks-on-change to upload to tinderbox-builds/%(branch)s-l10n and therefore we will only keep the latest copy (no dated dirs).
I have tested the nightly scenario and the x-testing repack on change scenario.

The problem was that we were not specifying the --tinderbox-builds-dir (which is probably caught by http://mxr.mozilla.org/build/source/tools/stage/post_upload.py#217 but we only get the return code)

Uploaded repack-on-change in here:
http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-l10n/

Comment 5

[Armen [:armenzg]]

The patch also tackles bug 509937

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 394821 [details] [diff] [review]
fixes the post_upload command used by repacks on change

(Armen asked me to review this with you gone this morning, Coop)

>diff --git a/process/factory.py b/process/factory.py
>--- a/process/factory.py
>+++ b/process/factory.py
>@@ -1430,24 +1430,29 @@
>         assert 'project' in kwargs
>         assert 'repoPath' in kwargs
>         if 'branchName' in kwargs:
>-          uploadDir = '%s-l10n' % kwargs['branchName']
>-        else:
>-          uploadDir = '%s-l10n' % self.getRepoName(kwargs['repoPath'])
>+          branchName = kwargs['branchName']
>+        else:
>+          branchName = self.getRepoName(kwargs['repoPath'])
>         self.env = {}
>         postUploadCmd = ['post_upload.py', 
>                          '-p %s ' % kwargs['project'],
>-                         '-b %s ' % uploadDir]
>+                         '-b %s-l10n ' % branchName]

Please don't move the -l10n part. Defining the uploadDir before constructing this command helps comprehension.

>         if l10nDatedDirs:
>-            postUploadCmd += ['-i %(buildid)s']
>+            # nightly repacks and on-change upload to different places
>             if self.nightly:
>-                postUploadCmd += ['--release-to-latest',
>+                postUploadCmd += ['--buildid %(buildid)s',
>+                                  '--release-to-latest',
>                                   '--release-to-dated']
>             else:
>                 # For the repack-on-change scenario we just want to upload
>-                # to tinderbox dated directories
>-                postUploadCmd += ['--release-to-tinderbox-dated-builds']
>+                # to tinderbox builds
>+                postUploadCmd += \
>+                      ['--tinderbox-builds-dir %s-l10n' % branchName,

Same thing here.


>             self.postUploadCmd = WithProperties(' '.join(postUploadCmd))
>         else:
>+            # for backwards compatibility when the nightly and repack on-change
>+            # runs where the same 

Grammar error, should be "were" not "where".

Comment 7

[Armen [:armenzg]]

Attachment: fixes the post_upload command used by repacks on change (v4)

Dealt with comments of bhearsum and tested a nightly run on staging-master

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 394852 [details] [diff] [review]
fixes the post_upload command used by repacks on change (v4)

changeset:   388:6de5a3f96d6e

Comment 9

[Armen [:armenzg]]

Both masters reconfigured and we are live!