1310229 - c-c/c-a l10n repack fails on calendar-upload due to incorrect buildid

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[aleth [:aleth]]

https://archive.mozilla.org/pub/thunderbird/nightly/2016/10/2016-10-14-07-45-47-comm-central-l10n/comm-central-linux64-l10n-nightly-sk-bm70-build1-build65.txt.gz

Executing: ['make', 'upload', 'AB_CD=sk']
mkdir -p `dirname '../../dist//thunderbird-52.0a1.sk.linux-x86_64.checksums'`
CHECKSUM FILE START
d2556deeba35e7e42674ebc04fedc8fad9d76e66dcf2e448444f5ac347c106a009ea849c6326a1ed486d484b8ff98ffa25880d24fe1dcffa69e09ef2992cdd82 sha512 54254614 thunderbird-52.0a1.sk.linux-x86_64.tar.bz2
f27bc7442dcf3603e9f3c2b52d22d674 md5 54254614 thunderbird-52.0a1.sk.linux-x86_64.tar.bz2
b739c537911965459c90887bb50446b07a7f13ea sha1 54254614 thunderbird-52.0a1.sk.linux-x86_64.tar.bz2
6083582619fb5b878de71404a0ed6fd15ad6ff48aca8791aa76fe451da87701e35ee211b1e8e6c73168be9f6d219f1018d6894c87c5e22ddfdb6a6fa8355ca4a sha512 54406694 update/thunderbird-52.0a1.sk.linux-x86_64.complete.mar
7bab769223be4ebc3562b70e035246ee md5 54406694 update/thunderbird-52.0a1.sk.linux-x86_64.complete.mar
2075da9639d66e0fc7516f134cde8d29ec795273 sha1 54406694 update/thunderbird-52.0a1.sk.linux-x86_64.complete.mar
a11e16f09ebf0cb1b05bc8dc8f237d4dc2b8d3b4b2db882af4d71f157beb5bc77c815306084c75ed77137730e19616064e150b4b34ae14b057685ceccb9681c6 sha512 582339 linux-x86_64/xpi/thunderbird-52.0a1.sk.langpack.xpi
aa63b11526cd20f19bacd682174ddabd md5 582339 linux-x86_64/xpi/thunderbird-52.0a1.sk.langpack.xpi
b12f5c8f3bcbc46ae3417b38458a135364318337 sha1 582339 linux-x86_64/xpi/thunderbird-52.0a1.sk.langpack.xpi
CHECKSUM FILE END
python /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/tools/release/signing/signtool.py --cachedir /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/signing_cache -t /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/token -n /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/nonce -c /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/tools/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9100 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9100 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9100 -f gpg '../../dist//thunderbird-52.0a1.sk.linux-x86_64.checksums'
2016-10-14 10:45:41,028 - 63a9a51dcbaa729c6f72e3eb4fa7aedb80903dcd: exists in the cache; copying to ../../dist//thunderbird-52.0a1.sk.linux-x86_64.checksums.asc
2016-10-14 10:45:41,028 - 63a9a51dcbaa729c6f72e3eb4fa7aedb80903dcd: OK
make -C ../../calendar/lightning upload AB_CD=sk
make[1]: Entering directory `/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tb/calendar/lightning'
/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tb/_virtualenv/bin/python /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/config/nsinstall.py -D ../../dist/linux-x86_64
/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tb/_virtualenv/bin/python /builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/config/nsinstall.py -R -m 644 ../../dist/xpi-stage/lightning-5.4a1.sk.linux-x86_64.xpi ../../dist/linux-x86_64
POST_UPLOAD_CMD="post_upload.py -b comm-central-l10n -p calendar/lightning -i Traceback (most recent call last):
/bin/sh: -c: line 0: unexpected EOF while looking for matching `"'
/bin/sh: -c: line 1: syntax error: unexpected end of file
make[1]: *** [upload-sk] Error 1
make[1]: Leaving directory `/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tb/calendar/lightning'
make: *** [calendar-upload] Error 2

Comment 1

[aleth [:aleth]]

Looks like this happens because POST_UPLOAD_CMD contains an error message:

POST_UPLOAD_CMD="post_upload.py -b comm-central-l10n -p thunderbird -i Traceback (most recent call last):\n  File "/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/mozilla/config/printconfigsetting.py", line 16, in <module>\n    with open(file) as fh:\nIOError: [Errno 2] No such file or directory: \'/builds/slave/tb-c-cen-l64-l10n-ntly-0000000/build/comm-central/objdir-tbdist/l10n-stage/thunderbird/application.ini\' --release-to-latest --release-to-dated"

Comment 2

[aleth [:aleth]]

And this happens because getting the buildid fails, and so the buildid is an error message (in factory.py::postUploadCmdPrefix).

Comment 3

[aleth [:aleth]]

On Windows it's 

POST_UPLOAD_CMD=post_upload.py -b comm-central-l10n -p thunderbird -i New python executable in c:\builds\moz2_slave\tb-c-cen-w32-l10n-ntly-0000000\build\comm-central\mozilla\obj-i686-pc-mingw32\_virtualenv\Scripts\python.exe
Installing setuptools, pip, wheel...done.
running build_ext

copying build\lib.win32-2.7\psutil\_psutil_windows.pyd -> psutil


c:\builds\moz2_slave\tb-c-cen-w32-l10n-ntly-0000000\build\comm-central\mozilla\python/mozbuild\mozbuild\virtualenv.py:376: UserWarning: Hacking environment to allow binary Python extensions to build. You can make this warning go away by installing Visual Studio 2008. You can download the Express Edition installer from http://go.microsoft.com/?linkid=7729279
  warnings.warn('Hacking environment to allow binary Python '
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-c-cen-w32-l10n-ntly-0000000/build/comm-central/mozilla/config/printconfigsetting.py", line 16, in <module>
    with open(file) as fh:
IOError: [Errno 2] No such file or directory: 'c:/builds/moz2_slave/tb-c-cen-w32-l10n-ntly-0000000/build/comm-central/objdir-tbdist/l10n-stage/thunderbird/application.ini' --release-to-latest --release-to-dated

Comment 4

[aleth [:aleth]]

NB One can already see the incorrect build id right at the top of the log file.

Comment 5

[aleth [:aleth]]

> NB One can already see the incorrect build id right at the top of the log
> file.

Interestingly, the build id was fine yesterday, e.g. in https://archive.mozilla.org/pub/thunderbird/nightly/2016/10/2016-10-14-07-45-47-comm-central-l10n/comm-central-linux64-l10n-nightly-sk-bm70-build1-build65.txt.gz

Comment 6

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #5)
> Interestingly, the build id was fine yesterday, e.g. in
> https://archive.mozilla.org/pub/thunderbird/nightly/2016/10/2016-10-14-07-45-
> 47-comm-central-l10n/comm-central-linux64-l10n-nightly-sk-bm70-build1-
> build65.txt.gz

That link should have been
https://ftp.mozilla.org/pub/thunderbird/nightly/2016/10/2016-10-13-14-04-22-comm-central-l10n/comm-central-win32-l10n-nightly-bn-BD-bm73-build1-build408.txt.gz

Comment 7

[aleth [:aleth]]

Probably NightlyRepackFactory.doRepack needs some changes.

Comment 8

[aleth [:aleth]]

Attachment: Fix incorrect path to application.ini in NightlyRepackFactory.doRepack printconfigsetting calls

The key change here is adding the missing / in the path to application.ini. I fixed the TOOLTOOL_DIR path used as well, though I don't think it has any effect here.

Comment 9

[aleth [:aleth]]

Attachment: Fix incorrect path to application.ini in NightlyRepackFactory.doRepack printconfigsetting calls

s/branchName/repackBranchName to avoid any potential collisions with an existing self.branchName.

Comment 10

[aleth [:aleth]]

Attachment: Fix incorrect path to application.ini in NightlyRepackFactory.doRepack printconfigsetting calls

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8801210 [details] [diff] [review]
Fix incorrect path to application.ini in NightlyRepackFactory.doRepack printconfigsetting calls

>diff --git a/process/factory.py b/process/factory.py
...
>         if 'branchName' in kwargs:
>             branchName = kwargs['branchName']
>         else:
>             branchName = self.getRepoName(kwargs['repoPath'])
>+        self.repackBranchName = branchName

It turns out that MercurialBuildFactory.__init__() has the same logic and sets self.branchName, see http://hg.mozilla.org/build/buildbotcustom/file/3b0c1d8cb4eb/process/factory.py#l443. So lets not bother setting self.repackBranchName here ...

>+        printconfig_env.update({
>+            'TOOLTOOL_DIR': WithProperties('%(basedir)s/build/' +
>+                                           self.repackBranchName)

... and use self.branchName here. I've tested it has the same result. r+ to land with those changes.

Comment 12

[aleth [:aleth]]

(In reply to Nick Thomas [:nthomas] from comment #11)
> It turns out that MercurialBuildFactory.__init__() has the same logic and
> sets self.branchName, see
> http://hg.mozilla.org/build/buildbotcustom/file/3b0c1d8cb4eb/process/factory.
> py#l443. So lets not bother setting self.repackBranchName here ...
> 
> >+        printconfig_env.update({
> >+            'TOOLTOOL_DIR': WithProperties('%(basedir)s/build/' +
> >+                                           self.repackBranchName)
> 
> ... and use self.branchName here. I've tested it has the same result. r+ to
> land with those changes.

For future reference, it's in MozillaBuildFactory.__init__; thanks!

Comment 13

[aleth [:aleth]]

https://hg.mozilla.org/build/buildbotcustom/rev/e0b271fa8cc4b00bf4c93a5f1aa410a7aec3ce34
Bug 1310229 - Fix incorrect path to application.ini in NightlyRepackFactory.doRepack printconfigsetting calls. r=nthomas

Comment 14

[aleth [:aleth]]

Merged to production at https://hg.mozilla.org/build/buildbotcustom/rev/0f38b7152713

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to aleth [:aleth] from comment #12)
> For future reference, it's in MozillaBuildFactory.__init__; thanks!

E_TOO_MANY_FACTORIES !

Comment 16

[aleth [:aleth]]

Looks like we're still in trouble on Windows: The build id is printed, but together with a bunch of other stuff.

https://ftp.mozilla.org/pub/thunderbird/nightly/2016/10/2016-10-19-16-04-58-comm-central-l10n/comm-central-win32-l10n-nightly-uk-bm74-build1-build374.txt.gz

New python executable in c:\builds\moz2_slave\tb-c-cen-w32-l10n-ntly-0000000\build\comm-central\mozilla\obj-i686-pc-mingw32\_virtualenv\Scripts\python.exe
Installing setuptools, pip, wheel...done.
running build_ext

copying build\lib.win32-2.7\psutil\_psutil_windows.pyd -> psutil


c:\builds\moz2_slave\tb-c-cen-w32-l10n-ntly-0000000\build\comm-central\mozilla\python/mozbuild\mozbuild\virtualenv.py:376: UserWarning: Hacking environment to allow binary Python extensions to build. You can make this warning go away by installing Visual Studio 2008. You can download the Express Edition installer from http://go.microsoft.com/?linkid=7729279
  warnings.warn('Hacking environment to allow binary Python '
20161019201458

Comment 17

[aleth [:aleth]]

There's no actual error... just the python install output. Since this is missing from the next printconfigsetting call (for the app version), another (hacky) way to fix this would be to simply set the build id twice in a row on Windows.

Is there a better way? Would using mock help here?

Comment 18

[aleth [:aleth]]

Attachment: Add dummy python call for Windows repacks to ensure it is installed before the first printconfigsetting call

Or something like this might work...

Comment 19

[aleth [:aleth]]

Attachment: Add dummy python call for Windows repacks to ensure it is installed before the first printconfigsetting call

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8802776 [details] [diff] [review]
Add dummy python call for Windows repacks to ensure it is installed before the first printconfigsetting call

I think I'd rather make sure MOZ_OBJDIR is set so the existing virutalenv is used, ie this one from the configure call:
c:/builds/moz2_slave/tb-c-cen-w32-l10n-ntly-0000000/build/comm-central/configure
Creating Python environment
New python executable in c:\builds\moz2_slave\tb-c-cen-w32-l10n-ntly-0000000\build\comm-central\objdir-tb\_virtualenv\Scripts\python2.7.exe

Comment 21

[aleth [:aleth]]

Attachment: Restore MOZ_OBJDIR in printconfigsetting environment for repacks

It's possible this will need an if(Windows) to not break something else. I don't know why that line was there.

Comment 22

[aleth [:aleth]]

Attachment: Restore MOZ_OBJDIR in printconfigsetting environment for repacks

I took the opportunity to also fix the broken workdir path.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Fix windows printconfigsetting.py calls

aleth, MOZ_OBJDIR was the wrong suggestion, sorry! Here's a tested fix instead - it turns out current call is almost right, except that the python on these windows machines ends up at _virtualenv/Scripts/python instead of the posix-style _virtualenv/bin/python.

rail - this affects TB l10n jobs only, which are failing to call printconfigsetting.py to extract buildid, app version and name. You may recall bug 1232466 from last year, where we ended up using the system python on windows to do a 'python mach python' call. This just changes it to use the python in the virtualenv which was created earlier in the job. It makes it like linux & mac, while allowing for the Windows oddness of having a Scripts/ instead of bin/.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

aleth, it turns out the workdir fix isn't a requirement, presumably because we're doing
 /abs/path/to/python /abs/path/to/mach /abs/path/to/printconfigsettings.py
so I left it out.

Comment 25

[aleth [:aleth]]

https://hg.mozilla.org/build/buildbotcustom/rev/1b8422e7e50adfa940b656339d5ae66f39af825e
Bug 1310229 - Correct the virtualenv python path for TB Windows repacks. r=rail

Comment 26

[aleth [:aleth]]

In production https://hg.mozilla.org/build/buildbotcustom/rev/9298bcec029fa9af7925314a4dcca04aec486d1b

Comment 27

[aleth [:aleth]]

(In reply to Nick Thomas [:nthomas] from comment #23)
> aleth, MOZ_OBJDIR was the wrong suggestion, sorry! Here's a tested fix
> instead - it turns out current call is almost right, except that the python
> on these windows machines ends up at _virtualenv/Scripts/python instead of
> the posix-style _virtualenv/bin/python.

Thanks!

I'm a bit surprised though that this works as I had thought the problem was with the second python in "python mach python", not the first.