Closed Bug 1286440 Opened 3 years ago Closed 3 years ago

Thunderbird 49.0b1 mac repacks failing

Categories

(Thunderbird :: Build Config, defect, blocker)

defect
Not set
blocker

Tracking

(thunderbird49 fixed, thunderbird50 fixed, thunderbird51 fixed)

RESOLVED FIXED
Thunderbird 49.0
Tracking Status
thunderbird49 --- fixed
thunderbird50 --- fixed
thunderbird51 --- fixed

People

(Reporter: nthomas, Assigned: ewong)

References

Details

Attachments

(6 files, 9 obsolete files)

2.87 KB, patch
jlund
: review+
Details | Diff | Splinter Review
2.78 KB, patch
jlund
: review+
Details | Diff | Splinter Review
742 bytes, patch
clokep
: review+
Details | Diff | Splinter Review
834 bytes, patch
aleth
: review+
Details | Diff | Splinter Review
1.44 KB, patch
Details | Diff | Splinter Review
959 bytes, patch
mkmelin
: review+
Details | Diff | Splinter Review
On linux64 in configure:
checking for the target C compiler... not found
DEBUG: _cc: Trying /builds/slave/tb-rel-c-beta-l64_rpk_10-00000/comm-beta/mozilla/gcc/bin/gcc
ERROR: Cannot find the target C compiler

tooltool has unpacked it at comm-beta/gcc/bin/gcc instead. Similarly PKG_CONFIG points to /builds/slave/tb-rel-c-beta-l64_rpk_10-00000/comm-beta/mozilla/gtk3/usr/local/lib/pkgconfig but the file is that less the mozilla/. eg https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-linux64_repack_1-bm74-build1-build2.txt.gz

Linux32 looks the same, eg https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-linux_repack_1-bm74-build1-build4.txt.gz

On mac it's
checking for the target C++ compiler... /usr/bin/clang++
checking whether the target C++ compiler can be used... no
...
DEBUG: | error: invalid value 'gnu++11' in '-std=gnu++11'

eg https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-macosx64_repack_1-bm84-build1-build19.txt.gz
I've filed this in Release Automation but it may be a build system issue. Tooltool is unpacking the archives in the same place  as 47.0b2 (..../comm-beta), and I don't think TOOLTOOL_DIR is being set in the environment, so we end up at 
  https://dxr.mozilla.org/mozilla-beta/source/build/unix/mozconfig.gtk#5
  https://dxr.mozilla.org/mozilla-beta/source/build/unix/mozconfig.linux#10

Has topsrcdir, or something in the comm build system, changed between 47 and 48 ? I am out of clues.
windows repacks 1,9,10 failed on retry. The other 7 succeeded
Thunderbird 48 mozconfig seems to include a bunch of environment variables, including TOOLTOOL_DIR=/builds/slave/tb-rel-c-beta-lx_rpk_1-0000000/comm-beta/mozilla 

Looking at the cwd for tooltool_wrapper.sh it runs in comm-beta, so I suspect the files are being downloaded/extracted to /builds/slave/tb-rel-c-beta-lx_rpk_1-0000000/comm-beta instead.

It would be interesting to know where those environment variables come from. Search for "Adding configure options" to get to the difference in configuration and search for "Fetching..." to get to the tooltool checkout in these logs:

Thunderbird 47:
https://archive.mozilla.org/pub/thunderbird/candidates/47.0b1-candidates/build1/logs/release-comm-beta-linux_repack_1-bm74-build1-build3.txt.gz

Thunderbird 48:
https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-linux_repack_1-bm74-build1-build4.txt.gz

Callek, I hope this makes it easier for you to take a look, thanks for doing so!
Flags: needinfo?(bugspam.Callek)
Sooo.. I think the TOOLTOOL issue described here is indeed whats wrong with linux repacks.

https://hg.mozilla.org/releases/comm-beta/rev/1e33571d13dc was a fix :ewong did for suite, something similar I suspect will work for TB.
Flags: needinfo?(bugspam.Callek)
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #2)
> windows repacks 1,9,10 failed on retry. The other 7 succeeded

I misstated - it should be win32 1,4,9,10.  4,9,10 completed today after callek retriggers.

callek commented about 1 "Timeout doing signing it seems, retried."
The retry failed "Status: exception"
The solution for that is to stick a TOOLTOOL_DIR=.. in the repack step
in the buildbotcustom code.
Attached patch [buildbotcustom] proposed patch (v1) (obsolete) β€” β€” Splinter Review
This basically adds the TOOLTOOL_DIR to the env during signing, which
will instruct the build config to use the basedir/build and not whatever
path it is in.
Attachment #8774009 - Flags: review?(jlund)
Attachment #8774009 - Flags: review?(jlund) → feedback?(jlund)
Attached patch [buildbotcustom] proposed patch (v2) (obsolete) β€” β€” Splinter Review
Having spoken with :jlund on irc and realizing how invasive the
v1 patch was, I've come up with a less invasive patch which
will only apply to Thunderbird.
Attachment #8774009 - Attachment is obsolete: true
Attachment #8774009 - Flags: feedback?(jlund)
Attachment #8774029 - Flags: review?(jlund)
Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +825,5 @@
>                              '--bucket-prefix', branchConfig.get('bucket_prefix'),
>                          ])
> +                    if 'mozilla_srcdir' in releaseConfig:
> +                        useBranch = releaseConfig['sourceRepositories'][sourceRepoKey]['name']
> +                        env.update({'TOOLTOOL_DIR': normalizeName(builddir, useBranch)})

I would prefer to replace these 3 lines to:

env.update(releaseConfig.get('extra_repack_env'), {})

then define 'extra_repack_env' in all of the buildbot-configs/mozilla/release-thunderbird-* configs with "TOOLTOOL_DIR" key.

that way we don't need a condition here in this patch.
Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +825,5 @@
>                              '--bucket-prefix', branchConfig.get('bucket_prefix'),
>                          ])
> +                    if 'mozilla_srcdir' in releaseConfig:
> +                        useBranch = releaseConfig['sourceRepositories'][sourceRepoKey]['name']
> +                        env.update({'TOOLTOOL_DIR': normalizeName(builddir, useBranch)})

also, this is sort of a hack. we don't know the slave's working path until runtime. so baking this in to the factory is fragile.
Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

one more thought, your first patch pointed TOOLTOOL_DIR to 'build'. this patch will point to either: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#57 or https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#45

which is the gecko repo root path...
Comment on attachment 8774009 [details] [diff] [review]
[buildbotcustom] proposed patch (v1)

Review of attachment 8774009 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +4594,5 @@
>                      property='toolsdir',
>                      workdir='scripts',
>                  ))
>              signing_env = self.env.copy()
> +            signing_env.update({'TOOLTOOL_DIR': WithProperties('%(basedir)s/build')}

is there a way we can use this patch. but condition a `if thunderbird` ?
Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

see comment 12 for suggestion.

ftr - I like your approach for limiting scope to just tb but my biggest concern is using hardcoded slave working paths
Attachment #8774029 - Flags: review?(jlund) → review-
(In reply to Jordan Lund (:jlund) from comment #12)
> Comment on attachment 8774009 [details] [diff] [review]
> [buildbotcustom] proposed patch (v1)
> 
> Review of attachment 8774009 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: process/factory.py
> @@ +4594,5 @@
> >                      property='toolsdir',
> >                      workdir='scripts',
> >                  ))
> >              signing_env = self.env.copy()
> > +            signing_env.update({'TOOLTOOL_DIR': WithProperties('%(basedir)s/build')}
> 
> is there a way we can use this patch. but condition a `if thunderbird` ?

Possibly.  I'm basically wondering if we can check if the mozillaSrcDir
kwarg is in the SigningScriptFactory or at least, pass an env var
into the SigningScriptFactory().  


>one more thought, your first patch pointed TOOLTOOL_DIR to 'build'. this patch will point to either: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#57 or https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#45

> which is the gecko repo root path...

That would be wrong since iiuc, the signing should be based on the
basedir + branch path.
Attached patch [configs] set tooltool_env (obsolete) β€” β€” Splinter Review
Attachment #8774565 - Flags: review?(jlund)
Attached patch [buildbotcustom] proposed patch (v3) (obsolete) β€” β€” Splinter Review
Attachment #8774029 - Attachment is obsolete: true
Attachment #8774566 - Flags: review?(jlund)
Comment on attachment 8774566 [details] [diff] [review]
[buildbotcustom] proposed patch (v3)

Review of attachment 8774566 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +4560,3 @@
>          self.signingServers = signingServers
>          self.enableSigning = enableSigning
> +        self.rel_config = rel_config

I don't think we want to pass in the whole releaseconfig. maybe extra_sign_env ?

so, with hindsight, maybe something like:

# from release config file
releaseConfig['extra_signing_env'] = {'TOOLTOOL_DIR': '%(basedir)s/comm-beta'}

# then in this file
self.extra_signing_env = extra_signing_env or {}
signing_env.update(
    {key, WithProperties(value) for key, value in self.extra_signing_env.iteritems()}
)

@@ +4598,5 @@
>                  ))
>              signing_env = self.env.copy()
>              signing_env['MOZ_SIGN_CMD'] = WithProperties(get_signing_cmd(
>                  self.signingServers, self.env.get('PYTHON26')))
> +            if 'tooltool_env' in self.rel_config:

don't need this condition

@@ +4599,5 @@
>              signing_env = self.env.copy()
>              signing_env['MOZ_SIGN_CMD'] = WithProperties(get_signing_cmd(
>                  self.signingServers, self.env.get('PYTHON26')))
> +            if 'tooltool_env' in self.rel_config:
> +                tmp_env = {key, WithProperties(value) for key, value in self.rel_config.get('tooltool_env', {}).iteritems()}

I gave you bad syntax in irc. dict comprehensions should be like {key: value for key value in some_dict.iteritems()}

so with the above comments you could:

signing_env.update({key: WithProperties(value) for key, value in self.extra_signing_env.iteritems()})
Attachment #8774566 - Flags: review?(jlund) → review-
Comment on attachment 8774565 [details] [diff] [review]
[configs] set tooltool_env

Review of attachment 8774565 [details] [diff] [review]:
-----------------------------------------------------------------

what branches does this also affect today? will this break as we uplift to release/esr?

in addition to this file, you will need to patch the following with the same line addition:

release-thunderbird-comm-beta.py.template
staging_release-thunderbird-comm-beta.py
staging_release-thunderbird-comm-beta.py.template

::: mozilla/release-thunderbird-comm-beta.py
@@ +165,5 @@
>  # Misc configuration
>  releaseConfig['enableAutomaticPushToMirrors'] = True
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('linux','linux64')
> +releaseConfig['tooltool_env'] = {'TOOLTOOL_DIR': '%(basedir)s/comm-beta'}

see comment 17 for review comments
Attachment #8774565 - Flags: review?(jlund) → review-
(In reply to Jordan Lund (:jlund) from comment #18)
> Comment on attachment 8774565 [details] [diff] [review]
> [configs] set tooltool_env
> 
> Review of attachment 8774565 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> what branches does this also affect today? will this break as we uplift to
> release/esr?

aiui, it affects the current beta.. and probably will affect the
next esr (so the new esr configs will need these lines as well)
Attachment #8774565 - Attachment is obsolete: true
Attachment #8774619 - Flags: review?(jlund)
Attached patch [buildbotcustom] proposed patch (v4) (obsolete) β€” β€” Splinter Review
Assignee: nobody → ewong
Attachment #8774566 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #8774620 - Flags: review?(jlund)
(In reply to Edmund Wong (:ewong) from comment #19)
> (In reply to Jordan Lund (:jlund) from comment #18)
> > Comment on attachment 8774565 [details] [diff] [review]
> > [configs] set tooltool_env
> > 
> > Review of attachment 8774565 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > what branches does this also affect today? will this break as we uplift to
> > release/esr?
> 
> aiui, it affects the current beta.. and probably will affect the
> next esr (so the new esr configs will need these lines as well)

okay, can you file a follow up patch for tracking esr work once it lands on esr?
Comment on attachment 8774620 [details] [diff] [review]
[buildbotcustom] proposed patch (v4)

Review of attachment 8774620 [details] [diff] [review]:
-----------------------------------------------------------------

almost! :)

::: process/factory.py
@@ +4598,5 @@
>                  ))
>              signing_env = self.env.copy()
>              signing_env['MOZ_SIGN_CMD'] = WithProperties(get_signing_cmd(
>                  self.signingServers, self.env.get('PYTHON26')))
> +            signing_env.update({key, WithProperties(value) for key, value in self.extra_signing_env.iteritems()})

s/key, WithProperties(value)/key: WithProperties(value)/

::: process/release.py
@@ +837,5 @@
>                          mock_packages=pf.get('mock_packages'),
>                          mock_copyin_files=pf.get('mock_copyin_files'),
>                          use_credentials_file=True,
>                          copy_properties=['buildid'],
> +                        rel_config=releaseConfig,

this should now be: extra_signing_env=releaseConfig.get("extra_signing_env")
Attachment #8774620 - Flags: review?(jlund) → review-
Comment on attachment 8774619 [details] [diff] [review]
[configs] proposed patch (v2)

Review of attachment 8774619 [details] [diff] [review]:
-----------------------------------------------------------------

looks great :)
Attachment #8774619 - Flags: review?(jlund) → review+
Attachment #8774620 - Attachment is obsolete: true
Attachment #8774992 - Flags: review?(jlund)
Comment on attachment 8774992 [details] [diff] [review]
[buildbotcustom] proposed patch (v5)

Review of attachment 8774992 [details] [diff] [review]:
-----------------------------------------------------------------

fingers crossed!
Attachment #8774992 - Flags: review?(jlund) → review+
possible to land?
We'd like to be building early next week in order to push out version 49 (At this point we've missed beta 48)
Flags: needinfo?(ewong)
https://hg.mozilla.org/build/buildbot-configs/rev/e2f6fb3d8204ec5162295c56a1bdaf872260d7a0
Bug 1286440 - Set TOOLTOOL_DIR env in SigningScriptFactory for comm-beta. r=jlund
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #27)
> possible to land?
> We'd like to be building early next week in order to push out version 49 (At
> this point we've missed beta 48)

Landed.  Just waiting for the travis ci status to go green.
Flags: needinfo?(ewong)
Did the repacks go well with the release?
Not sure of the relevancy  of this comment
We ditched beta48 but for beta 49
Linux repacks seemed to work OK But: 

https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build1/logs/release-comm-beta-win32_build-bm77-build1-build0.txt.gz

mozmake.exe[4]: *** [dom/base/target] Error 1073807364
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
c:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/client.mk:394: recipe for target 'build' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build'
mozmake.exe[1]: *** [build] Error 2 

and many mac repacks failing
IErepack_9/10 failed for Thunderbird 49.0b1 build1 on macosx64

Status: failure

Full details are available at:
 http://buildbot-master85.bb.releng.scl3.mozilla.com:8001/builders/release-comm-beta-macosx64_repack_9%2F10/builds/10

Releng has not fully investigated the build failure at this time
Great that the linux repacks working (modulo some machine disconnects and one mock install problem). The windows build is not related AFAICT, so lets not overload this bug further - I've rerun the job to see what happens.

For mac repacks it looks it's not finding the clang from tooltool, eg in configure
checking for the target C++ compiler... /usr/bin/clang++
checking whether the target C++ compiler can be used... DEBUG: <truncated - see config.log for full output>

/usr/bin/clang++ is old, v2.1.0. 

For comparison, the en-US build does:
checking for the target C++ compiler... /builds/slave/tb-rel-c-beta-m64_bld-00000000/build/clang/bin/clang++
checking whether the target C++ compiler can be used... yes
checking the target C++ compiler version... 3.8.0

They're both using the same tooltool manifest, which is used without issue, and TOOLTOOL_DIR appears to be set properly. Who wants to play spot-the-difference ?
I'm guessing it's something to do with universal builds using
 https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx-universal/release
Many includes inside mozilla-beta/build/macosx/universal/mozconfig there is
 https://dxr.mozilla.org/mozilla-beta/source/build/macosx/local-mozconfig.common#19
to set CC CXX and so on.

The l10n mozconfig at https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx-universal/l10n-mozconfig doesn't do anything similar.
Looks like 10n is actually using https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx64/l10n-mozconfig. We've reached the comm build system, who can help fix this ?
Philipp or Aleth perhaps?
Flags: needinfo?(philipp)
Flags: needinfo?(aleth)
Severity: normal → blocker
Component: Release Automation → Build Config
Product: Release Engineering → Thunderbird
Filed bug 1293401 for the windows build failures
(In reply to Nick Thomas [:nthomas] from comment #36)
> Looks like 10n is actually using
> https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx64/
> l10n-mozconfig. We've reached the comm build system, who can help fix this ?

I'll give it a try if no one else chimes in.
Attached patch [mozconfig] proposed patch (v1) (obsolete) β€” β€” Splinter Review
Attachment #8779143 - Flags: review?(Pidgeot18)
Comment on attachment 8779143 [details] [diff] [review]
[mozconfig] proposed patch (v1)

No... wrong file.
Attachment #8779143 - Attachment is obsolete: true
Attachment #8779143 - Flags: review?(Pidgeot18)
Attached patch [mozconfigs] proposed patch (v2) (obsolete) β€” β€” Splinter Review
This patch sets the proper CC/CXX env vars for the OSX l10n-mozconfig repack
build.
Attachment #8779152 - Flags: review?(Pidgeot18)
Are you sure that's the right file ? I think we don't use universal for repacks for other products, so it would be macosx64/l10n-mozconfig.
:nthomas sorry.. you're right.
Attachment #8779152 - Attachment is obsolete: true
Attachment #8779152 - Flags: review?(Pidgeot18)
Attachment #8779157 - Flags: review?(Pidgeot18)
Patch lgtm, let's try it? :-)
Flags: needinfo?(aleth)
Comment on attachment 8779157 [details] [diff] [review]
[mozconfigs] proposed patch (v3)

Seems reasonable to me and matches how we're doing Linux.
Attachment #8779157 - Flags: review?(Pidgeot18) → review+
Just wondering why some use:
. $topsrcdir/build/macosx/universal/mozconfig.common
and others use:
. $topsrcdir/build/macosx/universal/mozconfig
(In reply to Ian Neal from comment #47)
> Just wondering why some use:
> . $topsrcdir/build/macosx/universal/mozconfig.common
> and others use:
> . $topsrcdir/build/macosx/universal/mozconfig

Some builds (i.e. repacks) don't need both projects (i386 and x64),
so they call universal/mozconfig.common.  Builds that need both
projects (i.e. regular builds.. ) use universal/mozconfig.
Please, someone needs to check this in so we can get on with builds.
Thanks.

(not sure why fallen is still NI)
https://hg.mozilla.org/comm-central/rev/2c46af2ad50be236b77e527030854b87599590b6
Bug 1286440 - Get OSX64 repacks to detect clang from tooltool. r=jcranmer
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #49)
> Please, someone needs to check this in so we can get on with builds.
> Thanks.

You can just set checkin-needed for this.
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 51.0
If this needs uplift to beta (?) please set the required flags ;)
Flags: needinfo?(philipp) → needinfo?(vseerror)
Comment on attachment 8779157 [details] [diff] [review]
[mozconfigs] proposed patch (v3)

[Approval Request Comment]
Regression caused by (bug #): -
User impact if declined: can't ship builds
Testing completed (on c-c, etc.): only just landed on c-c, but per cloke "matches how we're doing Linux."
Risk to taking this patch (and alternatives if risky): Can't be more broken than it already is? :)  (and no current alternative)
Flags: needinfo?(vseerror)
Attachment #8779157 - Flags: approval-comm-beta?
Attachment #8779157 - Flags: approval-comm-aurora?
Attachment #8779157 - Flags: approval-comm-beta?
Attachment #8779157 - Flags: approval-comm-beta+
Attachment #8779157 - Flags: approval-comm-aurora?
Attachment #8779157 - Flags: approval-comm-aurora+
> Comment on attachment 8779157 [details] [diff] [review] [diff] [review]
> [mozconfigs] proposed patch (v3)

checkin-needed for c-beta and c-aurora
Keywords: checkin-needed
checkin-needed is for C-C only. I approved it, so I know it needs to land. I'm onto it, trust me (despite it being almost 1 AM).
Keywords: checkin-needed
every non-linux repack failed for 49.0b1 build2.
I have no time to investigate, unfortunately, to see what's going on.
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #58)
> every non-linux repack failed for 49.0b1 build2.
> I have no time to investigate, unfortunately, to see what's going on.

Are you running into Bug 1267523? Adding --disable-compile-environment to the l10n mozconfigs was the next thing we were going to test for that.
(In reply to aleth [:aleth] from comment #59)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #58)
> > every non-linux repack failed for 49.0b1 build2.
> > I have no time to investigate, unfortunately, to see what's going on.
> 
> Are you running into Bug 1267523? Adding --disable-compile-environment to
> the l10n mozconfigs was the next thing we were going to test for that.

see email containing build log
Flags: needinfo?(nthomas)
This passes configure on one of the machines that failed 49.0b1 build2. wsmwk, could you organise review and approval flags ?
Flags: needinfo?(nthomas) → needinfo?(vseerror)
Comment on attachment 8782665 [details] [diff] [review]
[mozconfigs] Use --with-l10n-base=../../../l10n

Review of attachment 8782665 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine - but I suspect the other l10n-mozconfig files will need the same change?

Thanks for the patch! It's hard to test l10n mozconfig changes locally.
Attachment #8782665 - Flags: review+
Comment on attachment 8782665 [details] [diff] [review]
[mozconfigs] Use --with-l10n-base=../../../l10n

[Approval Request Comment]
Testing completed (on c-c, etc.): see previous comment
Risk to taking this patch (and alternatives if risky): None
Flags: needinfo?(vseerror)
Attachment #8782665 - Flags: approval-comm-beta?
Attachment #8782665 - Flags: approval-comm-aurora?
M(In reply to aleth [:aleth] from comment #62)
> Looks fine - but I suspect the other l10n-mozconfig files will need the same
> change?

Maybe other mac mozconfigs, no idea what the state of other builds is.
Comment on attachment 8782665 [details] [diff] [review]
[mozconfigs] Use --with-l10n-base=../../../l10n

Aurora (TB 50):
https://hg.mozilla.org/releases/comm-aurora/rev/742127b0c0ea

Beta (TB 49):
https://hg.mozilla.org/releases/comm-beta/rev/d51efd110cb9

Wayne, for your build you now have to use changeset d51efd110cb9.
Attachment #8782665 - Flags: approval-comm-beta?
Attachment #8782665 - Flags: approval-comm-beta+
Attachment #8782665 - Flags: approval-comm-aurora?
Attachment #8782665 - Flags: approval-comm-aurora+
checking for pkg_config... not found
ERROR: Invalid value --with-l10n-base, ../../l10n doesn't exist
*** Fix above errors and then restart with               "make -f client.mk build"
make[2]: *** [configure] Error 1
make[1]: *** [configure] Error 2
make: *** [configure] Error 2
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/lib/python/util/commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
command: END (441.85s elapsed)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Summary: Thunderbird 48.0b1 repacks failing → Thunderbird 49.0b1 mac repacks failing
Did you start a new build with the new changeset?

../../l10n doesn't exist, well we've changed it to: ../../../l10n
(In reply to Jorg K (GMT+2, PTO during summer) from comment #68)
> Did you start a new build with the new changeset?
> 
> ../../l10n doesn't exist, well we've changed it to: ../../../l10n

yes. Quote from build automation email "Build of Thunderbird-49.0b1-build3"
....comm commit: https://hg.mozilla.org/releases/comm-beta/rev/d51efd110cb9


The "new" error in https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build3/logs/release-comm-beta-macosx64_repack_10-bm85-build1-build8.txt.gz is

  File "/tools/python27/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'
Please disregard comment 67 (I was looking and the wrong build log) sorry for the bugspam
Attachment #8783144 - Flags: review?(jlund)
Status: REOPENED → ASSIGNED
tl;dr

The rationale for the two patches is because (and I'm not even sure why it wasn't
busting prior to this):

How things are called:

from : http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l827

it calls release_repacks.sh with the given extra_args,

and release_repacks.sh does a bunch of stuff and then calls create_release_repacks.py with a bunch of options(some taken form
extra_args, others taken from the properties file..buildprops.json).

Now the nitty details (fyi):

http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l106

uses the objdir as set from http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l41,

which is taken from : http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l367

While the parser includes the parameter option"objdir" via:
http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l251

it isn't used in the call from within release_repacks.sh:
http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/release_repacks.sh#l73

So, it when create-release-repacks.py is called, it uses the default:
http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l243

While this works for Linux* and Win*, it doesn't work for Macs because
of the i386, x64 paths within the objdir.  (Because of the concept of
universal builds?)

The two patches included does the repacks against the objdir-tb/i386.

This is where I'm not sure because I don't know much about how mac
builds things and I don't know much about what happens within the
repacks.

So if someone can clarify if it is sufficient to repack against
the i386 configured tree, or does it also need to repack against
the x86-64 one?

If it needs to repack against the x86-64, I'm guessing there should be
a loop in the release_repacks.sh to run against both objdir-tb/i386
and objdir-tb/x86-64.

Now this all begs the question "Do my changes affect Firefox and mobile?"

For mobile, it uses a different 'branch' in the buildbotcustom
code: http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l780
(so no, unless I"ve mistaken/misread).

Unfortunately, I haven't been able to figure out whether Firefox uses
this part of the code due to the new release promotion and that the
logs are not easily found. (They're in taskcluster and I have no idea
how to find the logs to do any comparing.)

Any clarifications appreciated.
(In reply to Edmund Wong (:ewong) from comment #73)
> tl;dr
> 
> The rationale for the two patches is because (and I'm not even sure why it

*sigh*, I never completed this sentence..  

The rationale for the two patches is to explicitly set the objdir
in the SigningScriptFactory() call so that it propagates down to
create-release-repacks.py.
A way back at the clang issue, did you considering sourcing mozilla-beta/build/macosx/local-mozconfig.common ? Or even just duplicate these four lines into the l10n mozconfig
 https://dxr.mozilla.org/mozilla-beta/source/build/macosx/local-mozconfig.common#21-24

I haven't tried this, but it seems like a much more targeted fix than converting the l10n repacks to a universal build.
(In reply to Nick Thomas [:nthomas] from comment #75)
> A way back at the clang issue, did you considering sourcing
> mozilla-beta/build/macosx/local-mozconfig.common ? Or even just duplicate
> these four lines into the l10n mozconfig
>  https://dxr.mozilla.org/mozilla-beta/source/build/macosx/local-mozconfig.
> common#21-24
> 
> I haven't tried this, but it seems like a much more targeted fix than
> converting the l10n repacks to a universal build.

Hi Nick,

It didn't occur to me to do that because since the current setup
actually builds i386,x86-64 versions, I figured they'd need to
repack against either of the binaries or both.  If it's irrelevant
to do so, then, your solution makes sense.
Attached patch [mozconfig] l10n-config patch (obsolete) β€” β€” Splinter Review
From comment #75, this is my attempt at this patch.

If clang is in $topsrcdir/  then it's a mozilla-* build.
If clang is in $topsrcdir/../  then it's a comm-* build (non-universal mac)
if clang is in $topsrcdir/../../ then it's a comm-* build (universal mac)

:glandium, am I right?  Does this make sense?
Attachment #8783374 - Flags: feedback?(mh+mozilla)
Would setting --disable-compile-environment avoid the problem altogether?
Comment on attachment 8783374 [details] [diff] [review]
[mozconfig] l10n-config patch

Review of attachment 8783374 [details] [diff] [review]:
-----------------------------------------------------------------

What aleth says. Why do you need clang for a repack in the first place?
Attachment #8783374 - Flags: feedback?(mh+mozilla)
Attachment #8783144 - Attachment is obsolete: true
Attachment #8783146 - Attachment is obsolete: true
Attachment #8783374 - Attachment is obsolete: true
Attachment #8783144 - Flags: review?(jlund)
Attachment #8783146 - Flags: review?(jlund)
Attachment #8783759 - Flags: review?(mkmelin+mozilla)
Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

Review of attachment 8783759 [details] [diff] [review]:
-----------------------------------------------------------------

Thx, rs=mkmelin
Attachment #8783759 - Flags: review?(mkmelin+mozilla) → review+
Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: repacks fail
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky): none
Attachment #8783759 - Flags: approval-comm-beta?
Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

C-C: https://hg.mozilla.org/comm-central/rev/6bbdff770fd6
C-A: Required?
C-B: https://hg.mozilla.org/releases/comm-beta/rev/c047b4011b8c
Attachment #8783759 - Flags: approval-comm-beta? → approval-comm-beta+
(In reply to Jorg K (GMT+2, PTO during summer) from comment #83)
> Comment on attachment 8783759 [details] [diff] [review]
> [mozconfigs] disable compile environment
> 
> C-C: https://hg.mozilla.org/comm-central/rev/6bbdff770fd6
> C-A: Required?

Yes, otherwise it will be confusing when c-a becomes c-b...
Attachment #8783759 - Flags: approval-comm-aurora?
Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

C-A: https://hg.mozilla.org/releases/comm-aurora/rev/2ee34730d900
Attachment #8783759 - Flags: approval-comm-aurora? → approval-comm-aurora+
per nthomas "There's a typo in the l10n mozconfig change, should be --disable-compile-environment (it's missing the n in ...vironme...)"

so we need to reland the patches
Flags: needinfo?(jorgk)
Thanks jorg. LGTM
There now is a different repack failure blocking us - Mac repack log below from 49.0b1 build 4.
Is it deserving of a new bug report?
(linux repacks worked again. No sign yet of windows repacks)

https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build4/logs/release-comm-beta-macosx64_repack_1-bm84-build1-build20.txt.gz
command: START
command: make export
command: cwd: comm-beta/obj-l10n/config
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'MOZ_SIGN_CMD': 'python /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/signing_cache -t /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/token -n /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/nonce -c /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'UPLOAD_SSH_KEY': '~/.ssh/tbirdbld_dsa', 'COMM_REV': 'THUNDERBIRD_49_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'DOWNLOAD_HOST': 'archive.mozilla.org', 'UPLOAD_USER': 'tbirdbld', 'UPLOAD_HOST': 'upload.tbirdbld.productdelivery.prod.mozaws.net', 'POST_UPLOAD_CMD': 'post_upload.py -p thunderbird -n 4 -v 49.0b1 --release-to-candidates-dir --signed --bucket-prefix net-mozaws-prod-delivery', 'MOZILLA_REV': 'THUNDERBIRD_49_0b1_RELEASE', 'MOZ_PKG_VERSION': '49.0b1', 'MBSDIFF_HOOK': '/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/scripts/l10n/mbsdiff_hook.sh -c /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/fs-cache', 'MOZ_PKG_PRETTYNAMES': '1'}
command: output:
command: END (0.08s elapsed)

Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 394, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/lib/python/build/l10n.py", line 100, in l10nRepackPrep
    env=env)
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/lib/python/util/commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 506, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/tools/python27/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/tools/python27/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'
program finished with exit code 1
elapsedTime=628.490386
(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #89)
> OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'
> program finished with exit code 1
> elapsedTime=628.490386

This looks like Comment 69. Maybe some of ewong's proposed changes for this are needed?
Flags: needinfo?(ewong)
it seems that windows are failing too but for a different reason: tooltool

ERROR - The following files failed: 'vs2015u2.zip'
retry: Giving up on <function run_with_timeout at 0x02831C70>
Unable to successfully run ['python.exe', 'c:/mozilla-build/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', 'mail/config/tooltool-manifests/win32/releng.manifest', 'fetch'] after 10 attempts
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['sh', '../scripts/scripts/tooltool/tooltool_wrapper.sh', 'mail/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python', 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
command: END (1747.77s elapsed)

Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/create-release-repacks.py", line 394, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\build\l10n.py", line 80, in l10nRepackPrep
    run_cmd(cmd, cwd=sourceRepoName)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sh', '../scripts/scripts/tooltool/tooltool_wrapper.sh', 'mail/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python', 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
program finished with exit code 1
elapsedTime=1906.894000

we probably want to open up a meta tracking tb 49.0b1 bug and file a bug for new error in comment 89 and one for this windows issue. This bug has already tracked a lot of issues.
Blocks: 1297787
(In reply to aleth [:aleth] from comment #90)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #89)
> > OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'
> > program finished with exit code 1
> > elapsedTime=628.490386
> 
> This looks like Comment 69. Maybe some of ewong's proposed changes for this
> are needed?

I'll reinstate those proposed patches and see if jlund or anyone else has
a comment.
Flags: needinfo?(ewong)
Comment on attachment 8783146 [details] [diff] [review]
[buildbotcustom] include l10n_objdir in the call to release_repacks.sh

reinstating for :jlund's review/comment, as per comment #90.
Attachment #8783146 - Attachment is obsolete: false
Attachment #8783146 - Flags: review?(jlund)
Attachment #8783144 - Attachment is obsolete: false
Attachment #8783144 - Flags: review?(jlund)
Comment on attachment 8783144 [details] [diff] [review]
[tools] add objdir parameter to release_repacks.sh

Moving this to bug 1297786
Attachment #8783144 - Attachment is obsolete: true
Attachment #8783144 - Flags: review?(jlund)
Comment on attachment 8783146 [details] [diff] [review]
[buildbotcustom] include l10n_objdir in the call to release_repacks.sh

Moving to bug 1297786
Attachment #8783146 - Flags: review?(jlund)
(In reply to Jordan Lund (:jlund) from comment #91)
> it seems that windows are failing too but for a different reason: tooltool
> 
> ERROR - The following files failed: 'vs2015u2.zip'
> retry: Giving up on <function run_with_timeout at 0x02831C70>
> Unable to successfully run ['python.exe', 'c:/mozilla-build/tooltool.py',
> '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m',
> 'mail/config/tooltool-manifests/win32/releng.manifest', 'fetch'] after 10
> attempts
> command: ERROR
> Traceback (most recent call last):
>   File
> "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-
> 000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
>     return subprocess.check_call(cmd, **kwargs)
>   File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
>     raise CalledProcessError(retcode, cmd)
> CalledProcessError: Command '['sh',
> '../scripts/scripts/tooltool/tooltool_wrapper.sh',
> 'mail/config/tooltool-manifests/win32/releng.manifest',
> 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python',
> 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
> command: END (1747.77s elapsed)
> 
> Traceback (most recent call last):
>   File
> "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/
> create-release-repacks.py", line 394, in <module>
>     bucket_prefix=branchConfig['bucket_prefix'],
>   File
> "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/
> create-release-repacks.py", line 108, in createRepacks
>     makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
>   File
> "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-
> 000000\scripts\lib\python\build\l10n.py", line 80, in l10nRepackPrep
>     run_cmd(cmd, cwd=sourceRepoName)
>   File
> "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-
> 000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
>     return subprocess.check_call(cmd, **kwargs)
>   File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['sh',
> '../scripts/scripts/tooltool/tooltool_wrapper.sh',
> 'mail/config/tooltool-manifests/win32/releng.manifest',
> 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python',
> 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
> program finished with exit code 1
> elapsedTime=1906.894000
> 
> we probably want to open up a meta tracking tb 49.0b1 bug and file a bug for
> new error in comment 89 and one for this windows issue. This bug has already
> tracked a lot of issues.

I'm wondering if this is an intermittent issue?
That's handled on bug 1297785, the visual studio file from tooltool needs an token to get access.
Remainder fixed in the followup bug 1297786.
Status: ASSIGNED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: Thunderbird 51.0 → Thunderbird 49.0
looking back with this bug, can someone point out why we need to
add --disable-compile-environment to mac repacks whereas the other
platforms (that I can tell) don't have this/need this?  Is it because
of universal build?  Even participating in this bug, and looking back,
I don't understand this.  (It also doesn't help that I did a mess
with the patches, so I can't follow any of the logic.)
(In reply to Edmund Wong (:ewong) from comment #99)
> looking back with this bug, can someone point out why we need to
> add --disable-compile-environment to mac repacks whereas the other
> platforms (that I can tell) don't have this/need this?  Is it because
> of universal build?  Even participating in this bug, and looking back,
> I don't understand this.  (It also doesn't help that I did a mess
> with the patches, so I can't follow any of the logic.)

iirc see comment 77 to 79, one can either make sure a number of compiler variables are set correctly, or make them irrelevant by disabling the compile environment. Either approach works. (In bug 1267523 I was initially hoping disable-compile-environment would also mean we wouldn't have to set TOOLTOOL_DIR, but that was mistaken. And still needs to be done for c-c/c-a repacks :-( )
You need to log in before you can comment on or make changes to this bug.