1346020 - Thunderbird 52.0 repacks failing

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Wayne Mery (:wsmwk)]

We're not doing well

https://archive.mozilla.org/pub/thunderbird/candidates/52.0-candidates/build1/logs/release-comm-esr52-win32_repack_1-bm72-build1-build0.txt.gz

command: output:
a15a29e4a07c
Updated to revision a15a29e4a07c2d5182985adf47b54d177a33cf34.
Warning: Preferring MOZILLA_REV (THUNDERBIRD_52_0_RELEASE) over passed ...
  File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\util.py", line 943, in __get__
    setattr(instance, name, self.func(instance))
  File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\configure\__init__.py", line 90, in result
    return self.func(*resolved_args)
  File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\configure\__init__.py", line 932, in wrapped
    return new_func(*args, **kwargs)
  File "c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/mozilla/build/moz.configure/init.configure", line 119, in mozconfig
    mozconfig = loader.read_mozconfig(mozconfig, moz_build_app=current_project)
  File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\comm-esr52\mozilla\python\mozbuild\mozbuild\mozconfig.py", line 257, in read_mozconfig
    raise MozconfigLoadException(path, MOZCONFIG_BAD_EXIT_CODE, lines)
mozbuild.mozconfig.MozconfigLoadException: Evaluation of your mozconfig exited with an error. This could be triggered
by a command inside your mozconfig failing. Please change your mozconfig
to not error and/or to catch errors in executed commands.
*** Fix above errors and then restart with\

"c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/mozmake.exe -f client.mk build"
c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/client.mk:360: recipe for target 'configure' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52'
mozmake.exe[1]: *** [configure] Error 1
client.mk:217: recipe for target 'configure' failed
mozmake.exe: *** [configure] Error 2
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-esr52-w32_rpk_1-00000\\comm-esr52/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
command: END (194.68s elapsed)

Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/scripts/scripts/l10n/create-release-repacks.py", line 384, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\scripts\lib\python\build\l10n.py", line 85, in l10nRepackPrep
    env=env)
  File "c:\builds\moz2_slave\tb-rel-c-esr52-w32_rpk_1-00000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['c:\\mozilla-build\\python27\\python.exe', 'c:\\builds\\moz2_slave\\tb-rel-c-esr52-w32_rpk_1-00000\\comm-esr52/build/pymake/make.py', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
program finished with exit code 1


On 3/9/2017 4:57 PM, release--- via release-automation-notifications-thunderbird wrote:
> repack_1/10 failed for Thunderbird 52.0 build1 on win32
>
> Status: failure
>
> Full details are available at:
>   http://buildbot-master72.bb.releng.usw2.mozilla.com:8001/builders/release-comm-esr52-win32_repack_1%2F10/builds/0
>
> -buildbot

Comment 1

[Wayne Mery (:wsmwk)]

Mac repacks did not fail.
FWIW, windows nightly repacks also failing bug 1345422

Comment 2

[Wayne Mery (:wsmwk)]

further up, is this normal?

mkdir -p 'c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/'
client.mk:203: c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/.mozconfig.mk: No such file or directory

Comment 3

[Wayne Mery (:wsmwk)]

(blocking release)

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #2)
> further up, is this normal?
> 
> mkdir -p
> 'c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/'
> client.mk:203:
> c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/obj-l10n/.
> mozconfig.mk: No such file or directory
That's also in the log for the repack of 52.0b4: https://archive.mozilla.org/pub/thunderbird/candidates/52.0b4-candidates/build1/logs/release-comm-beta-win32_repack_1-bm91-build1-build3.txt.gz
The first line missing in the release log compare to the one for 52.0b4 is
>   TOOLTOOL_DIR=/c/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/comm-beta

Next significant different seems to be:

beta:
>python2.7 c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_1-000000/comm-beta/client.py checkout --hg-options='--time' --hgtool=../tools/buildfarm/utils/hgtool.py --hgtool1=../scripts/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --tinderbox-print

release:
>python2.7 c:/builds/moz2_slave/tb-rel-c-esr52-w32_rpk_1-00000/comm-esr52/client.py checkout --hg-options='--time' --hgtool=../tools/buildfarm/utils/hgtool.py --hgtool1=../scripts/buildfarm/utils/hgtool.py --skip-chatzilla --skip-comm --skip-inspector --tinderbox-print --mozilla-repo=https://hg.mozilla.org/releases/mozilla-esr52 --mozilla-rev=THUNDERBIRD_52_VERBRANCH

See https://dxr.mozilla.org/comm-central/rev/d03f6d1f69374c85d4939d0c7cb92021614600f4/client.py#445

Down the line:
>Warning: Preferring MOZILLA_REV (THUNDERBIRD_52_0_RELEASE) over passed revision (THUNDERBIRD_52_VERBRANCH)

Comment 5

[Philipp Kewisch [:Fallen][📱📆]]

This looks like the ESR version of bug 1286440. Reading bug 1286440 comment 100 it is still not clear to me why we are not consistent with --disable-compile-environment.

I propose we do the same buildbot-configs fix as in that bug, plus add --disable-compile-environment to all win/linux mozconfigs.

aleth, can you confirm this is ok?

Comment 6

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: buildbot-configs changes - v1

Here is a patch for the buildbot-configs part. I've also taken the liberty to copy the missing staging config from the esr45 files, if this is not needed anymore or the simple s/45/52/ doesn't quite do it please let me know.

Comment 7

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: mozconfig changes - v1

Here are the mozconfig changes. I created the patch with a comm-esr52 repo, but we should probably push this to all repos once we have confirmed it works for 52.

Comment 8

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 8848267 [details] [diff] [review]
mozconfig changes - v1

Review of attachment 8848267 [details] [diff] [review]:
-----------------------------------------------------------------

Jörg, if you have insight into this issue and aleth is not available I am happy to take your review.

Comment 9

[Jorg K (CEST = GMT+2)]

Comment on attachment 8848267 [details] [diff] [review]
mozconfig changes - v1

(Greetings from Catalonia where my phone/internet still work ;-))
I have dug around in bug 1322402 disabling universal builds, and I was surprised about the inconsistencies with respect to this flag.

Reading through bug 1286440 comment #77 downwards and also reading bug 301197 (which first introduced the flag) I think the proposed change here is right, but we should make sure we fix all branches once TB 52 builds fine (I can do that). Besides, it's not such a drastic change.

So r=jorgk when you get the buildbot part approved.

Thanks for the patches, BTW.

Comment 10

[Mike Hommey [:glandium]]

Comment on attachment 8848266 [details] [diff] [review]
buildbot-configs changes - v1

Review of attachment 8848266 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not a buildbot peer, sorry.

Comment 11

[Edmund Wong (:ewong)]

:Fallen, might it work if you include the following line to
l10n-mozconfig:

. "$topsrcdir/build/mozconfig.win-common"

Comment 12

[Philipp Kewisch [:Fallen][📱📆]]

ewong, the patches I created were based on reading the linked bug, I wasn't actually able to verify it locally. I don't have a non-mac machine set up for testing at the moment, so I also cannot test if including mozconfig.win-common helps. Can you clarify why you believe that would fix it? Would this be instead of what I proposed, or in addition?

Comment 13

[Edmund Wong (:ewong)]

(In reply to Philipp Kewisch [:Fallen] from comment #12)
> ewong, the patches I created were based on reading the linked bug, I wasn't
> actually able to verify it locally. I don't have a non-mac machine set up
> for testing at the moment, so I also cannot test if including
> mozconfig.win-common helps. Can you clarify why you believe that would fix
> it? Would this be instead of what I proposed, or in addition?

Hi Fallen,

I apologize for adding my $0.02 when in fact, the issue is the basic
topsrcdir != expected topsrcdir.

I've tried replicating the repack on a loaner; but the way the repacks
are done, it isn't the easiest thing to do namely because of all the
processes being lumped under the create-release-repacks.py script.

That said, while I'm still having some problems getting past a certain
point, the issue I think with the current MOZCONFIG bustage is that
you'll need your buildbot-config patch.  It's another one of those
topsrcdir isn't the right topsrcdir that is expected in the mozconfig
loading.

Particularly this line:
+releaseConfig['extra_signing_env'] = {'TOOLTOOL_DIR': '%(basedir)s/comm-esr52'}

As for the mozconfig patch, my suggestion was based on ignorance. 
Mind you, with our buildbot config patch, I don't think the --disable-compile-environment is necessary; but now that I've looked at how TB does its repacks,
it could be a different thing.  So my humblest apologies.


particularly it was (at least in my loaner env) trying to access
the VS2015u3 path.

'c:/builds/slave/tbtmp/comm-esr52/mozilla/build/win32/mozconfig.vs2015-win64: line 6: cd: /c/builds/slave/tbtmp/comm-esr52/mozilla/vs2015u3: No such file or directory'

I believe that it's sufficient with the buildbot-configs patch.

Comment 14

[Philipp Kewisch [:Fallen][📱📆]]

ewong, no worries at all! I appreciate your input very much, I just had trouble understanding why it was needed and asked for clarification. I'm sorry if this came across blunt.

If the disable-compile-environment patch doesn't hurt I think it would be good for consistency. I hope the buildbot-configs changes will work as expected.

If you are successful testing with both patches applied that would be very helpful!

Comment 15

[Philipp Kewisch [:Fallen][📱📆]]

https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675cee83236ed49e0cf8013ce712e40 (default branch)

Comment 16

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675

Comment 17

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675

Comment 18

[Jorg K (CEST = GMT+2)]

Landing this on all branches, here comes C-C:
https://hg.mozilla.org/comm-central/rev/0d9f3ed365b7912b6e57fcd21ad11de536f3384c

Comment 19

[Jorg K (CEST = GMT+2)]

C-A (TB 54): https://hg.mozilla.org/releases/comm-aurora/rev/48de8f8524b74f4867bcc5b647e4c72e0e10291a
C-B (TB 53): https://hg.mozilla.org/releases/comm-beta/rev/32087f2e8d4e72e1765d0d9a5dd9c1a6adc466bc

Comment 20

[Jorg K (CEST = GMT+2)]

ESR 52: https://hg.mozilla.org/releases/comm-esr52/rev/a730476809a84b349be34db4b59013288ebc2803

Comment 21

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5fc070519675

Comment 22

[Wayne Mery (:wsmwk)]

Fallen, I assume you no longer aleth feedback. If that is incorrect please flag someone else.