1286440 - Thunderbird 49.0b1 mac repacks failing

Status: RESOLVED FIXED

(Thunderbird :: Build Config, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

On linux64 in configure:
checking for the target C compiler... not found
DEBUG: _cc: Trying /builds/slave/tb-rel-c-beta-l64_rpk_10-00000/comm-beta/mozilla/gcc/bin/gcc
ERROR: Cannot find the target C compiler

tooltool has unpacked it at comm-beta/gcc/bin/gcc instead. Similarly PKG_CONFIG points to /builds/slave/tb-rel-c-beta-l64_rpk_10-00000/comm-beta/mozilla/gtk3/usr/local/lib/pkgconfig but the file is that less the mozilla/. eg https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-linux64_repack_1-bm74-build1-build2.txt.gz

Linux32 looks the same, eg https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-linux_repack_1-bm74-build1-build4.txt.gz

On mac it's
checking for the target C++ compiler... /usr/bin/clang++
checking whether the target C++ compiler can be used... no
...
DEBUG: | error: invalid value 'gnu++11' in '-std=gnu++11'

eg https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-macosx64_repack_1-bm84-build1-build19.txt.gz

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I've filed this in Release Automation but it may be a build system issue. Tooltool is unpacking the archives in the same place  as 47.0b2 (..../comm-beta), and I don't think TOOLTOOL_DIR is being set in the environment, so we end up at 
  https://dxr.mozilla.org/mozilla-beta/source/build/unix/mozconfig.gtk#5
  https://dxr.mozilla.org/mozilla-beta/source/build/unix/mozconfig.linux#10

Has topsrcdir, or something in the comm build system, changed between 47 and 48 ? I am out of clues.

Comment 2

[Wayne Mery (:wsmwk)]

windows repacks 1,9,10 failed on retry. The other 7 succeeded

Comment 3

[Philipp Preißer-Kewisch [:Fallen][☁️📆]]

Thunderbird 48 mozconfig seems to include a bunch of environment variables, including TOOLTOOL_DIR=/builds/slave/tb-rel-c-beta-lx_rpk_1-0000000/comm-beta/mozilla 

Looking at the cwd for tooltool_wrapper.sh it runs in comm-beta, so I suspect the files are being downloaded/extracted to /builds/slave/tb-rel-c-beta-lx_rpk_1-0000000/comm-beta instead.

It would be interesting to know where those environment variables come from. Search for "Adding configure options" to get to the difference in configuration and search for "Fetching..." to get to the tooltool checkout in these logs:

Thunderbird 47:
https://archive.mozilla.org/pub/thunderbird/candidates/47.0b1-candidates/build1/logs/release-comm-beta-linux_repack_1-bm74-build1-build3.txt.gz

Thunderbird 48:
https://archive.mozilla.org/pub/thunderbird/candidates/48.0b1-candidates/build3/logs/release-comm-beta-linux_repack_1-bm74-build1-build4.txt.gz

Callek, I hope this makes it easier for you to take a look, thanks for doing so!

Comment 4

[Justin Wood (:Callek)]

Sooo.. I think the TOOLTOOL issue described here is indeed whats wrong with linux repacks.

https://hg.mozilla.org/releases/comm-beta/rev/1e33571d13dc was a fix :ewong did for suite, something similar I suspect will work for TB.

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #2)
> windows repacks 1,9,10 failed on retry. The other 7 succeeded

I misstated - it should be win32 1,4,9,10.  4,9,10 completed today after callek retriggers.

callek commented about 1 "Timeout doing signing it seems, retried."
The retry failed "Status: exception"

Comment 6

[Edmund Wong (:ewong)]

The solution for that is to stick a TOOLTOOL_DIR=.. in the repack step
in the buildbotcustom code.

Comment 7

[Edmund Wong (:ewong)]

Attachment: [buildbotcustom] proposed patch (v1)

This basically adds the TOOLTOOL_DIR to the env during signing, which
will instruct the build config to use the basedir/build and not whatever
path it is in.

Comment 8

[Edmund Wong (:ewong)]

Attachment: [buildbotcustom] proposed patch (v2)

Having spoken with :jlund on irc and realizing how invasive the
v1 patch was, I've come up with a less invasive patch which
will only apply to Thunderbird.

Comment 9

[Jordan Lund (:jlund)]

Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +825,5 @@
>                              '--bucket-prefix', branchConfig.get('bucket_prefix'),
>                          ])
> +                    if 'mozilla_srcdir' in releaseConfig:
> +                        useBranch = releaseConfig['sourceRepositories'][sourceRepoKey]['name']
> +                        env.update({'TOOLTOOL_DIR': normalizeName(builddir, useBranch)})

I would prefer to replace these 3 lines to:

env.update(releaseConfig.get('extra_repack_env'), {})

then define 'extra_repack_env' in all of the buildbot-configs/mozilla/release-thunderbird-* configs with "TOOLTOOL_DIR" key.

that way we don't need a condition here in this patch.

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/release.py
@@ +825,5 @@
>                              '--bucket-prefix', branchConfig.get('bucket_prefix'),
>                          ])
> +                    if 'mozilla_srcdir' in releaseConfig:
> +                        useBranch = releaseConfig['sourceRepositories'][sourceRepoKey]['name']
> +                        env.update({'TOOLTOOL_DIR': normalizeName(builddir, useBranch)})

also, this is sort of a hack. we don't know the slave's working path until runtime. so baking this in to the factory is fragile.

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

one more thought, your first patch pointed TOOLTOOL_DIR to 'build'. this patch will point to either: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#57 or https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#45

which is the gecko repo root path...

Comment 12

[Jordan Lund (:jlund)]

Comment on attachment 8774009 [details] [diff] [review]
[buildbotcustom] proposed patch (v1)

Review of attachment 8774009 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +4594,5 @@
>                      property='toolsdir',
>                      workdir='scripts',
>                  ))
>              signing_env = self.env.copy()
> +            signing_env.update({'TOOLTOOL_DIR': WithProperties('%(basedir)s/build')}

is there a way we can use this patch. but condition a `if thunderbird` ?

Comment 13

[Jordan Lund (:jlund)]

Comment on attachment 8774029 [details] [diff] [review]
[buildbotcustom] proposed patch (v2)

Review of attachment 8774029 [details] [diff] [review]:
-----------------------------------------------------------------

see comment 12 for suggestion.

ftr - I like your approach for limiting scope to just tb but my biggest concern is using hardcoded slave working paths

Comment 14

[Edmund Wong (:ewong)]

(In reply to Jordan Lund (:jlund) from comment #12)
> Comment on attachment 8774009 [details] [diff] [review]
> [buildbotcustom] proposed patch (v1)
> 
> Review of attachment 8774009 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: process/factory.py
> @@ +4594,5 @@
> >                      property='toolsdir',
> >                      workdir='scripts',
> >                  ))
> >              signing_env = self.env.copy()
> > +            signing_env.update({'TOOLTOOL_DIR': WithProperties('%(basedir)s/build')}
> 
> is there a way we can use this patch. but condition a `if thunderbird` ?

Possibly.  I'm basically wondering if we can check if the mozillaSrcDir
kwarg is in the SigningScriptFactory or at least, pass an env var
into the SigningScriptFactory().  


>one more thought, your first patch pointed TOOLTOOL_DIR to 'build'. this patch will point to either: https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#57 or https://dxr.mozilla.org/build-central/source/buildbot-configs/mozilla/release-thunderbird-comm-beta.py.template#45

> which is the gecko repo root path...

That would be wrong since iiuc, the signing should be based on the
basedir + branch path.

Comment 15

[Edmund Wong (:ewong)]

Attachment: [configs] set tooltool_env

Comment 16

[Edmund Wong (:ewong)]

Attachment: [buildbotcustom] proposed patch (v3)

Comment 17

[Jordan Lund (:jlund)]

Comment on attachment 8774566 [details] [diff] [review]
[buildbotcustom] proposed patch (v3)

Review of attachment 8774566 [details] [diff] [review]:
-----------------------------------------------------------------

::: process/factory.py
@@ +4560,3 @@
>          self.signingServers = signingServers
>          self.enableSigning = enableSigning
> +        self.rel_config = rel_config

I don't think we want to pass in the whole releaseconfig. maybe extra_sign_env ?

so, with hindsight, maybe something like:

# from release config file
releaseConfig['extra_signing_env'] = {'TOOLTOOL_DIR': '%(basedir)s/comm-beta'}

# then in this file
self.extra_signing_env = extra_signing_env or {}
signing_env.update(
    {key, WithProperties(value) for key, value in self.extra_signing_env.iteritems()}
)

@@ +4598,5 @@
>                  ))
>              signing_env = self.env.copy()
>              signing_env['MOZ_SIGN_CMD'] = WithProperties(get_signing_cmd(
>                  self.signingServers, self.env.get('PYTHON26')))
> +            if 'tooltool_env' in self.rel_config:

don't need this condition

@@ +4599,5 @@
>              signing_env = self.env.copy()
>              signing_env['MOZ_SIGN_CMD'] = WithProperties(get_signing_cmd(
>                  self.signingServers, self.env.get('PYTHON26')))
> +            if 'tooltool_env' in self.rel_config:
> +                tmp_env = {key, WithProperties(value) for key, value in self.rel_config.get('tooltool_env', {}).iteritems()}

I gave you bad syntax in irc. dict comprehensions should be like {key: value for key value in some_dict.iteritems()}

so with the above comments you could:

signing_env.update({key: WithProperties(value) for key, value in self.extra_signing_env.iteritems()})

Comment 18

[Jordan Lund (:jlund)]

Comment on attachment 8774565 [details] [diff] [review]
[configs] set tooltool_env

Review of attachment 8774565 [details] [diff] [review]:
-----------------------------------------------------------------

what branches does this also affect today? will this break as we uplift to release/esr?

in addition to this file, you will need to patch the following with the same line addition:

release-thunderbird-comm-beta.py.template
staging_release-thunderbird-comm-beta.py
staging_release-thunderbird-comm-beta.py.template

::: mozilla/release-thunderbird-comm-beta.py
@@ +165,5 @@
>  # Misc configuration
>  releaseConfig['enableAutomaticPushToMirrors'] = True
>  releaseConfig['use_mock'] = True
> +releaseConfig['mock_platforms'] = ('linux','linux64')
> +releaseConfig['tooltool_env'] = {'TOOLTOOL_DIR': '%(basedir)s/comm-beta'}

see comment 17 for review comments

Comment 19

[Edmund Wong (:ewong)]

(In reply to Jordan Lund (:jlund) from comment #18)
> Comment on attachment 8774565 [details] [diff] [review]
> [configs] set tooltool_env
> 
> Review of attachment 8774565 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> what branches does this also affect today? will this break as we uplift to
> release/esr?

aiui, it affects the current beta.. and probably will affect the
next esr (so the new esr configs will need these lines as well)

Comment 20

[Edmund Wong (:ewong)]

Attachment: [configs] proposed patch (v2)

Comment 21

[Edmund Wong (:ewong)]

Attachment: [buildbotcustom] proposed patch (v4)

Comment 22

[Jordan Lund (:jlund)]

(In reply to Edmund Wong (:ewong) from comment #19)
> (In reply to Jordan Lund (:jlund) from comment #18)
> > Comment on attachment 8774565 [details] [diff] [review]
> > [configs] set tooltool_env
> > 
> > Review of attachment 8774565 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > what branches does this also affect today? will this break as we uplift to
> > release/esr?
> 
> aiui, it affects the current beta.. and probably will affect the
> next esr (so the new esr configs will need these lines as well)

okay, can you file a follow up patch for tracking esr work once it lands on esr?

Comment 23

[Jordan Lund (:jlund)]

Comment on attachment 8774620 [details] [diff] [review]
[buildbotcustom] proposed patch (v4)

Review of attachment 8774620 [details] [diff] [review]:
-----------------------------------------------------------------

almost! :)

::: process/factory.py
@@ +4598,5 @@
>                  ))
>              signing_env = self.env.copy()
>              signing_env['MOZ_SIGN_CMD'] = WithProperties(get_signing_cmd(
>                  self.signingServers, self.env.get('PYTHON26')))
> +            signing_env.update({key, WithProperties(value) for key, value in self.extra_signing_env.iteritems()})

s/key, WithProperties(value)/key: WithProperties(value)/

::: process/release.py
@@ +837,5 @@
>                          mock_packages=pf.get('mock_packages'),
>                          mock_copyin_files=pf.get('mock_copyin_files'),
>                          use_credentials_file=True,
>                          copy_properties=['buildid'],
> +                        rel_config=releaseConfig,

this should now be: extra_signing_env=releaseConfig.get("extra_signing_env")

Comment 24

[Jordan Lund (:jlund)]

Comment on attachment 8774619 [details] [diff] [review]
[configs] proposed patch (v2)

Review of attachment 8774619 [details] [diff] [review]:
-----------------------------------------------------------------

looks great :)

Comment 25

[Edmund Wong (:ewong)]

Attachment: [buildbotcustom] proposed patch (v5)

Comment 26

[Jordan Lund (:jlund)]

Comment on attachment 8774992 [details] [diff] [review]
[buildbotcustom] proposed patch (v5)

Review of attachment 8774992 [details] [diff] [review]:
-----------------------------------------------------------------

fingers crossed!

Comment 27

[Wayne Mery (:wsmwk)]

possible to land?
We'd like to be building early next week in order to push out version 49 (At this point we've missed beta 48)

Comment 28

[Edmund Wong (:ewong)]

https://hg.mozilla.org/build/buildbot-configs/rev/e2f6fb3d8204ec5162295c56a1bdaf872260d7a0
Bug 1286440 - Set TOOLTOOL_DIR env in SigningScriptFactory for comm-beta. r=jlund

Comment 29

[Edmund Wong (:ewong)]

https://hg.mozilla.org/build/buildbotcustom/rev/c400d63b386dcfe91030e5706fe51fc0e5992142
Bug 1286440 - Set TOOLTOOL_DIR env in SigningScriptFactory. r=jlund

Comment 30

[Edmund Wong (:ewong)]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #27)
> possible to land?
> We'd like to be building early next week in order to push out version 49 (At
> this point we've missed beta 48)

Landed.  Just waiting for the travis ci status to go green.

Comment 31

[Nick Thomas [:nthomas] (UTC+12)]

Merged to production

> https://hg.mozilla.org/build/buildbot-configs/rev/e2f6fb3d8204ec5162295c56a1bdaf872260d7a0
https://hg.mozilla.org/build/buildbot-configs/rev/f1e84827aa8b

> https://hg.mozilla.org/build/buildbotcustom/rev/c400d63b386dcfe91030e5706fe51fc0e5992142
https://hg.mozilla.org/build/buildbotcustom/rev/61610b08ca2d

Comment 32

[Edmund Wong (:ewong)]

Did the repacks go well with the release?

Comment 33

[Joe Sabash [:JoeS1]]

Not sure of the relevancy  of this comment
We ditched beta48 but for beta 49
Linux repacks seemed to work OK But: 

https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build1/logs/release-comm-beta-win32_build-bm77-build1-build0.txt.gz

mozmake.exe[4]: *** [dom/base/target] Error 1073807364
mozmake.exe[3]: *** [compile] Error 2
mozmake.exe[2]: *** [default] Error 2
c:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build/client.mk:394: recipe for target 'build' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-rel-c-beta-w32_bld-00000000/build'
mozmake.exe[1]: *** [build] Error 2 

and many mac repacks failing
IErepack_9/10 failed for Thunderbird 49.0b1 build1 on macosx64

Status: failure

Full details are available at:
 http://buildbot-master85.bb.releng.scl3.mozilla.com:8001/builders/release-comm-beta-macosx64_repack_9%2F10/builds/10

Releng has not fully investigated the build failure at this time

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Great that the linux repacks working (modulo some machine disconnects and one mock install problem). The windows build is not related AFAICT, so lets not overload this bug further - I've rerun the job to see what happens.

For mac repacks it looks it's not finding the clang from tooltool, eg in configure
checking for the target C++ compiler... /usr/bin/clang++
checking whether the target C++ compiler can be used... DEBUG: <truncated - see config.log for full output>

/usr/bin/clang++ is old, v2.1.0. 

For comparison, the en-US build does:
checking for the target C++ compiler... /builds/slave/tb-rel-c-beta-m64_bld-00000000/build/clang/bin/clang++
checking whether the target C++ compiler can be used... yes
checking the target C++ compiler version... 3.8.0

They're both using the same tooltool manifest, which is used without issue, and TOOLTOOL_DIR appears to be set properly. Who wants to play spot-the-difference ?

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

I'm guessing it's something to do with universal builds using
 https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx-universal/release
Many includes inside mozilla-beta/build/macosx/universal/mozconfig there is
 https://dxr.mozilla.org/mozilla-beta/source/build/macosx/local-mozconfig.common#19
to set CC CXX and so on.

The l10n mozconfig at https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx-universal/l10n-mozconfig doesn't do anything similar.

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Looks like 10n is actually using https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx64/l10n-mozconfig. We've reached the comm build system, who can help fix this ?

Comment 37

[Jorg K (CEST = GMT+2)]

Philipp or Aleth perhaps?

Comment 38

[Joe Sabash [:JoeS1]]

Filed bug 1293401 for the windows build failures

Comment 39

[Edmund Wong (:ewong)]

(In reply to Nick Thomas [:nthomas] from comment #36)
> Looks like 10n is actually using
> https://dxr.mozilla.org/comm-beta/source/mail/config/mozconfigs/macosx64/
> l10n-mozconfig. We've reached the comm build system, who can help fix this ?

I'll give it a try if no one else chimes in.

Comment 40

[Edmund Wong (:ewong)]

Attachment: [mozconfig] proposed patch (v1)

Comment 41

[Edmund Wong (:ewong)]

Comment on attachment 8779143 [details] [diff] [review]
[mozconfig] proposed patch (v1)

No... wrong file.

Comment 42

[Edmund Wong (:ewong)]

Attachment: [mozconfigs] proposed patch (v2)

This patch sets the proper CC/CXX env vars for the OSX l10n-mozconfig repack
build.

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

Are you sure that's the right file ? I think we don't use universal for repacks for other products, so it would be macosx64/l10n-mozconfig.

Comment 44

[Edmund Wong (:ewong)]

Attachment: [mozconfigs] proposed patch (v3)

:nthomas sorry.. you're right.

Comment 45

[aleth [:aleth]]

Patch lgtm, let's try it? :-)

Comment 46

[Patrick Cloke [:clokep]]

Comment on attachment 8779157 [details] [diff] [review]
[mozconfigs] proposed patch (v3)

Seems reasonable to me and matches how we're doing Linux.

Comment 47

[Ian Neal]

Just wondering why some use:
. $topsrcdir/build/macosx/universal/mozconfig.common
and others use:
. $topsrcdir/build/macosx/universal/mozconfig

Comment 48

[Edmund Wong (:ewong)]

(In reply to Ian Neal from comment #47)
> Just wondering why some use:
> . $topsrcdir/build/macosx/universal/mozconfig.common
> and others use:
> . $topsrcdir/build/macosx/universal/mozconfig

Some builds (i.e. repacks) don't need both projects (i386 and x64),
so they call universal/mozconfig.common.  Builds that need both
projects (i.e. regular builds.. ) use universal/mozconfig.

Comment 49

[Wayne Mery (:wsmwk)]

Please, someone needs to check this in so we can get on with builds.
Thanks.

(not sure why fallen is still NI)

Comment 50

[aleth [:aleth]]

https://hg.mozilla.org/comm-central/rev/2c46af2ad50be236b77e527030854b87599590b6
Bug 1286440 - Get OSX64 repacks to detect clang from tooltool. r=jcranmer

Comment 51

[aleth [:aleth]]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #49)
> Please, someone needs to check this in so we can get on with builds.
> Thanks.

You can just set checkin-needed for this.

Comment 52

[aleth [:aleth]]

If this needs uplift to beta (?) please set the required flags ;)

Comment 53

[Wayne Mery (:wsmwk)]

Comment on attachment 8779157 [details] [diff] [review]
[mozconfigs] proposed patch (v3)

[Approval Request Comment]
Regression caused by (bug #): -
User impact if declined: can't ship builds
Testing completed (on c-c, etc.): only just landed on c-c, but per cloke "matches how we're doing Linux."
Risk to taking this patch (and alternatives if risky): Can't be more broken than it already is? :)  (and no current alternative)

Comment 54

[Wayne Mery (:wsmwk)]

> Comment on attachment 8779157 [details] [diff] [review] [diff] [review]
> [mozconfigs] proposed patch (v3)

checkin-needed for c-beta and c-aurora

Comment 55

[Jorg K (CEST = GMT+2)]

checkin-needed is for C-C only. I approved it, so I know it needs to land. I'm onto it, trust me (despite it being almost 1 AM).

Comment 56

[Jorg K (CEST = GMT+2)]

Aurora (TB 50):
https://hg.mozilla.org/releases/comm-aurora/rev/f2a07268a25c

Comment 57

[Jorg K (CEST = GMT+2)]

Beta (TB 49):
https://hg.mozilla.org/releases/comm-beta/rev/0b64fb78a07b

Comment 58

[Wayne Mery (:wsmwk)]

every non-linux repack failed for 49.0b1 build2.
I have no time to investigate, unfortunately, to see what's going on.

Comment 59

[aleth [:aleth]]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #58)
> every non-linux repack failed for 49.0b1 build2.
> I have no time to investigate, unfortunately, to see what's going on.

Are you running into Bug 1267523? Adding --disable-compile-environment to the l10n mozconfigs was the next thing we were going to test for that.

Comment 60

[Wayne Mery (:wsmwk)]

(In reply to aleth [:aleth] from comment #59)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #58)
> > every non-linux repack failed for 49.0b1 build2.
> > I have no time to investigate, unfortunately, to see what's going on.
> 
> Are you running into Bug 1267523? Adding --disable-compile-environment to
> the l10n mozconfigs was the next thing we were going to test for that.

see email containing build log

Comment 61

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [mozconfigs] Use --with-l10n-base=../../../l10n

This passes configure on one of the machines that failed 49.0b1 build2. wsmwk, could you organise review and approval flags ?

Comment 62

[aleth [:aleth]]

Comment on attachment 8782665 [details] [diff] [review]
[mozconfigs] Use --with-l10n-base=../../../l10n

Review of attachment 8782665 [details] [diff] [review]:
-----------------------------------------------------------------

Looks fine - but I suspect the other l10n-mozconfig files will need the same change?

Thanks for the patch! It's hard to test l10n mozconfig changes locally.

Comment 63

[aleth [:aleth]]

Comment on attachment 8782665 [details] [diff] [review]
[mozconfigs] Use --with-l10n-base=../../../l10n

[Approval Request Comment]
Testing completed (on c-c, etc.): see previous comment
Risk to taking this patch (and alternatives if risky): None

Comment 64

[aleth [:aleth]]

https://hg.mozilla.org/comm-central/rev/d1960937860dbe784e21b43ce136253fc70e77e2
Bug 1286440 - Use --with-l10n-base=../../../l10n. r=aleth

Comment 65

[Nick Thomas [:nthomas] (UTC+12)]

M(In reply to aleth [:aleth] from comment #62)
> Looks fine - but I suspect the other l10n-mozconfig files will need the same
> change?

Maybe other mac mozconfigs, no idea what the state of other builds is.

Comment 66

[Jorg K (CEST = GMT+2)]

Comment on attachment 8782665 [details] [diff] [review]
[mozconfigs] Use --with-l10n-base=../../../l10n

Aurora (TB 50):
https://hg.mozilla.org/releases/comm-aurora/rev/742127b0c0ea

Beta (TB 49):
https://hg.mozilla.org/releases/comm-beta/rev/d51efd110cb9

Wayne, for your build you now have to use changeset d51efd110cb9.

Comment 67

[Joe Sabash [:JoeS1]]

checking for pkg_config... not found
ERROR: Invalid value --with-l10n-base, ../../l10n doesn't exist
*** Fix above errors and then restart with               "make -f client.mk build"
make[2]: *** [configure] Error 1
make[1]: *** [configure] Error 2
make: *** [configure] Error 2
command: ERROR
Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/lib/python/util/commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['make', '-f', 'client.mk', 'configure']' returned non-zero exit status 2
command: END (441.85s elapsed)

Comment 68

[Jorg K (CEST = GMT+2)]

Did you start a new build with the new changeset?

../../l10n doesn't exist, well we've changed it to: ../../../l10n

Comment 69

[Wayne Mery (:wsmwk)]

(In reply to Jorg K (GMT+2, PTO during summer) from comment #68)
> Did you start a new build with the new changeset?
> 
> ../../l10n doesn't exist, well we've changed it to: ../../../l10n

yes. Quote from build automation email "Build of Thunderbird-49.0b1-build3"
....comm commit: https://hg.mozilla.org/releases/comm-beta/rev/d51efd110cb9


The "new" error in https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build3/logs/release-comm-beta-macosx64_repack_10-bm85-build1-build8.txt.gz is

  File "/tools/python27/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'

Comment 70

[Joe Sabash [:JoeS1]]

Please disregard comment 67 (I was looking and the wrong build log) sorry for the bugspam

Comment 71

[Edmund Wong (:ewong)]

Attachment: [tools] add objdir parameter to release_repacks.sh

Comment 72

[Edmund Wong (:ewong)]

Attachment: [buildbotcustom] include l10n_objdir in the call to release_repacks.sh

Comment 73

[Edmund Wong (:ewong)]

tl;dr

The rationale for the two patches is because (and I'm not even sure why it wasn't
busting prior to this):

How things are called:

from : http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l827

it calls release_repacks.sh with the given extra_args,

and release_repacks.sh does a bunch of stuff and then calls create_release_repacks.py with a bunch of options(some taken form
extra_args, others taken from the properties file..buildprops.json).

Now the nitty details (fyi):

http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l106

uses the objdir as set from http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l41,

which is taken from : http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l367

While the parser includes the parameter option"objdir" via:
http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l251

it isn't used in the call from within release_repacks.sh:
http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/release_repacks.sh#l73

So, it when create-release-repacks.py is called, it uses the default:
http://hg.mozilla.org/build/tools/file/tip/scripts/l10n/create-release-repacks.py#l243

While this works for Linux* and Win*, it doesn't work for Macs because
of the i386, x64 paths within the objdir.  (Because of the concept of
universal builds?)

The two patches included does the repacks against the objdir-tb/i386.

This is where I'm not sure because I don't know much about how mac
builds things and I don't know much about what happens within the
repacks.

So if someone can clarify if it is sufficient to repack against
the i386 configured tree, or does it also need to repack against
the x86-64 one?

If it needs to repack against the x86-64, I'm guessing there should be
a loop in the release_repacks.sh to run against both objdir-tb/i386
and objdir-tb/x86-64.

Now this all begs the question "Do my changes affect Firefox and mobile?"

For mobile, it uses a different 'branch' in the buildbotcustom
code: http://hg.mozilla.org/build/buildbotcustom/file/tip/process/release.py#l780
(so no, unless I"ve mistaken/misread).

Unfortunately, I haven't been able to figure out whether Firefox uses
this part of the code due to the new release promotion and that the
logs are not easily found. (They're in taskcluster and I have no idea
how to find the logs to do any comparing.)

Any clarifications appreciated.

Comment 74

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #73)
> tl;dr
> 
> The rationale for the two patches is because (and I'm not even sure why it

*sigh*, I never completed this sentence..  

The rationale for the two patches is to explicitly set the objdir
in the SigningScriptFactory() call so that it propagates down to
create-release-repacks.py.

Comment 75

[Nick Thomas [:nthomas] (UTC+12)]

A way back at the clang issue, did you considering sourcing mozilla-beta/build/macosx/local-mozconfig.common ? Or even just duplicate these four lines into the l10n mozconfig
 https://dxr.mozilla.org/mozilla-beta/source/build/macosx/local-mozconfig.common#21-24

I haven't tried this, but it seems like a much more targeted fix than converting the l10n repacks to a universal build.

Comment 76

[Edmund Wong (:ewong)]

(In reply to Nick Thomas [:nthomas] from comment #75)
> A way back at the clang issue, did you considering sourcing
> mozilla-beta/build/macosx/local-mozconfig.common ? Or even just duplicate
> these four lines into the l10n mozconfig
>  https://dxr.mozilla.org/mozilla-beta/source/build/macosx/local-mozconfig.
> common#21-24
> 
> I haven't tried this, but it seems like a much more targeted fix than
> converting the l10n repacks to a universal build.

Hi Nick,

It didn't occur to me to do that because since the current setup
actually builds i386,x86-64 versions, I figured they'd need to
repack against either of the binaries or both.  If it's irrelevant
to do so, then, your solution makes sense.

Comment 77

[Edmund Wong (:ewong)]

Attachment: [mozconfig] l10n-config patch

From comment #75, this is my attempt at this patch.

If clang is in $topsrcdir/  then it's a mozilla-* build.
If clang is in $topsrcdir/../  then it's a comm-* build (non-universal mac)
if clang is in $topsrcdir/../../ then it's a comm-* build (universal mac)

:glandium, am I right?  Does this make sense?

Comment 78

[aleth [:aleth]]

Would setting --disable-compile-environment avoid the problem altogether?

Comment 79

[Mike Hommey [:glandium]]

Comment on attachment 8783374 [details] [diff] [review]
[mozconfig] l10n-config patch

Review of attachment 8783374 [details] [diff] [review]:
-----------------------------------------------------------------

What aleth says. Why do you need clang for a repack in the first place?

Comment 80

[Edmund Wong (:ewong)]

Attachment: [mozconfigs] disable compile environment

Comment 81

[Magnus Melin [:mkmelin]]

Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

Review of attachment 8783759 [details] [diff] [review]:
-----------------------------------------------------------------

Thx, rs=mkmelin

Comment 82

[Edmund Wong (:ewong)]

Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

[Approval Request Comment]
Regression caused by (bug #): 
User impact if declined: repacks fail
Testing completed (on c-c, etc.): 
Risk to taking this patch (and alternatives if risky): none

Comment 83

[Jorg K (CEST = GMT+2)]

Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

C-C: https://hg.mozilla.org/comm-central/rev/6bbdff770fd6
C-A: Required?
C-B: https://hg.mozilla.org/releases/comm-beta/rev/c047b4011b8c

Comment 84

[aleth [:aleth]]

(In reply to Jorg K (GMT+2, PTO during summer) from comment #83)
> Comment on attachment 8783759 [details] [diff] [review]
> [mozconfigs] disable compile environment
> 
> C-C: https://hg.mozilla.org/comm-central/rev/6bbdff770fd6
> C-A: Required?

Yes, otherwise it will be confusing when c-a becomes c-b...

Comment 85

[Jorg K (CEST = GMT+2)]

Comment on attachment 8783759 [details] [diff] [review]
[mozconfigs] disable compile environment

C-A: https://hg.mozilla.org/releases/comm-aurora/rev/2ee34730d900

Comment 86

[Wayne Mery (:wsmwk)]

per nthomas "There's a typo in the l10n mozconfig change, should be --disable-compile-environment (it's missing the n in ...vironme...)"

so we need to reland the patches

Comment 87

[Jorg K (CEST = GMT+2)]

C-B: https://hg.mozilla.org/releases/comm-beta/rev/2b29b68058ea
C-A: https://hg.mozilla.org/releases/comm-aurora/rev/9ee09fdd13c5
C-C: https://hg.mozilla.org/comm-central/rev/5407fdff464f

Comment 88

[Wayne Mery (:wsmwk)]

Thanks jorg. LGTM

Comment 89

[Wayne Mery (:wsmwk)]

There now is a different repack failure blocking us - Mac repack log below from 49.0b1 build 4.
Is it deserving of a new bug report?
(linux repacks worked again. No sign yet of windows repacks)

https://archive.mozilla.org/pub/thunderbird/candidates/49.0b1-candidates/build4/logs/release-comm-beta-macosx64_repack_1-bm84-build1-build20.txt.gz
command: START
command: make export
command: cwd: comm-beta/obj-l10n/config
command: env: {'MOZ_MAKE_COMPLETE_MAR': '1', 'MOZ_SIGN_CMD': 'python /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/release/signing/signtool.py --cachedir /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/signing_cache -t /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/token -n /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/nonce -c /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9120 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9120', 'UPLOAD_SSH_KEY': '~/.ssh/tbirdbld_dsa', 'COMM_REV': 'THUNDERBIRD_49_0b1_RELEASE', 'MOZ_OBJDIR': 'obj-l10n', 'UPLOAD_TO_TEMP': '1', 'LD_LIBRARY_PATH': '', 'DOWNLOAD_HOST': 'archive.mozilla.org', 'UPLOAD_USER': 'tbirdbld', 'UPLOAD_HOST': 'upload.tbirdbld.productdelivery.prod.mozaws.net', 'POST_UPLOAD_CMD': 'post_upload.py -p thunderbird -n 4 -v 49.0b1 --release-to-candidates-dir --signed --bucket-prefix net-mozaws-prod-delivery', 'MOZILLA_REV': 'THUNDERBIRD_49_0b1_RELEASE', 'MOZ_PKG_VERSION': '49.0b1', 'MBSDIFF_HOOK': '/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/scripts/l10n/mbsdiff_hook.sh -c /builds/slave/tb-rel-c-beta-m64_rpk_1-000000/fs-cache', 'MOZ_PKG_PRETTYNAMES': '1'}
command: output:
command: END (0.08s elapsed)

Traceback (most recent call last):
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 394, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/lib/python/build/l10n.py", line 100, in l10nRepackPrep
    env=env)
  File "/builds/slave/tb-rel-c-beta-m64_rpk_1-000000/scripts/lib/python/util/commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 506, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/tools/python27/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/tools/python27/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/tools/python27/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'
program finished with exit code 1
elapsedTime=628.490386

Comment 90

[aleth [:aleth]]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #89)
> OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'
> program finished with exit code 1
> elapsedTime=628.490386

This looks like Comment 69. Maybe some of ewong's proposed changes for this are needed?

Comment 91

[Jordan Lund (:jlund)]

it seems that windows are failing too but for a different reason: tooltool

ERROR - The following files failed: 'vs2015u2.zip'
retry: Giving up on <function run_with_timeout at 0x02831C70>
Unable to successfully run ['python.exe', 'c:/mozilla-build/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', 'mail/config/tooltool-manifests/win32/releng.manifest', 'fetch'] after 10 attempts
command: ERROR
Traceback (most recent call last):
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['sh', '../scripts/scripts/tooltool/tooltool_wrapper.sh', 'mail/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python', 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
command: END (1747.77s elapsed)

Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/create-release-repacks.py", line 394, in <module>
    bucket_prefix=branchConfig['bucket_prefix'],
  File "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/create-release-repacks.py", line 108, in createRepacks
    makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\build\l10n.py", line 80, in l10nRepackPrep
    run_cmd(cmd, cwd=sourceRepoName)
  File "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
    return subprocess.check_call(cmd, **kwargs)
  File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sh', '../scripts/scripts/tooltool/tooltool_wrapper.sh', 'mail/config/tooltool-manifests/win32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python', 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
program finished with exit code 1
elapsedTime=1906.894000

we probably want to open up a meta tracking tb 49.0b1 bug and file a bug for new error in comment 89 and one for this windows issue. This bug has already tracked a lot of issues.

Comment 92

[Edmund Wong (:ewong)]

(In reply to aleth [:aleth] from comment #90)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #89)
> > OSError: [Errno 2] No such file or directory: 'comm-beta/obj-l10n/config'
> > program finished with exit code 1
> > elapsedTime=628.490386
> 
> This looks like Comment 69. Maybe some of ewong's proposed changes for this
> are needed?

I'll reinstate those proposed patches and see if jlund or anyone else has
a comment.

Comment 93

[Edmund Wong (:ewong)]

Comment on attachment 8783146 [details] [diff] [review]
[buildbotcustom] include l10n_objdir in the call to release_repacks.sh

reinstating for :jlund's review/comment, as per comment #90.

Comment 94

[Edmund Wong (:ewong)]

Comment on attachment 8783144 [details] [diff] [review]
[tools] add objdir parameter to release_repacks.sh

Moving this to bug 1297786

Comment 95

[Edmund Wong (:ewong)]

Comment on attachment 8783146 [details] [diff] [review]
[buildbotcustom] include l10n_objdir in the call to release_repacks.sh

Moving to bug 1297786

Comment 96

[Edmund Wong (:ewong)]

(In reply to Jordan Lund (:jlund) from comment #91)
> it seems that windows are failing too but for a different reason: tooltool
> 
> ERROR - The following files failed: 'vs2015u2.zip'
> retry: Giving up on <function run_with_timeout at 0x02831C70>
> Unable to successfully run ['python.exe', 'c:/mozilla-build/tooltool.py',
> '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m',
> 'mail/config/tooltool-manifests/win32/releng.manifest', 'fetch'] after 10
> attempts
> command: ERROR
> Traceback (most recent call last):
>   File
> "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-
> 000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
>     return subprocess.check_call(cmd, **kwargs)
>   File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
>     raise CalledProcessError(retcode, cmd)
> CalledProcessError: Command '['sh',
> '../scripts/scripts/tooltool/tooltool_wrapper.sh',
> 'mail/config/tooltool-manifests/win32/releng.manifest',
> 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python',
> 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
> command: END (1747.77s elapsed)
> 
> Traceback (most recent call last):
>   File
> "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/
> create-release-repacks.py", line 394, in <module>
>     bucket_prefix=branchConfig['bucket_prefix'],
>   File
> "c:/builds/moz2_slave/tb-rel-c-beta-w32_rpk_2-000000/scripts/scripts/l10n/
> create-release-repacks.py", line 108, in createRepacks
>     makeDirs, env, tooltoolManifest, tooltool_script, tooltool_urls)
>   File
> "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-
> 000000\scripts\lib\python\build\l10n.py", line 80, in l10nRepackPrep
>     run_cmd(cmd, cwd=sourceRepoName)
>   File
> "c:\builds\moz2_slave\tb-rel-c-beta-w32_rpk_2-
> 000000\scripts\lib\python\util\commands.py", line 52, in run_cmd
>     return subprocess.check_call(cmd, **kwargs)
>   File "c:\mozilla-build\python27\lib\subprocess.py", line 542, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['sh',
> '../scripts/scripts/tooltool/tooltool_wrapper.sh',
> 'mail/config/tooltool-manifests/win32/releng.manifest',
> 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', 'python',
> 'c:/mozilla-build/tooltool.py']' returned non-zero exit status 1
> program finished with exit code 1
> elapsedTime=1906.894000
> 
> we probably want to open up a meta tracking tb 49.0b1 bug and file a bug for
> new error in comment 89 and one for this windows issue. This bug has already
> tracked a lot of issues.

I'm wondering if this is an intermittent issue?

Comment 97

[Nick Thomas [:nthomas] (UTC+12)]

That's handled on bug 1297785, the visual studio file from tooltool needs an token to get access.

Comment 98

[aleth [:aleth]]

Remainder fixed in the followup bug 1297786.

Comment 99

[Edmund Wong (:ewong)]

looking back with this bug, can someone point out why we need to
add --disable-compile-environment to mac repacks whereas the other
platforms (that I can tell) don't have this/need this?  Is it because
of universal build?  Even participating in this bug, and looking back,
I don't understand this.  (It also doesn't help that I did a mess
with the patches, so I can't follow any of the logic.)

Comment 100

[aleth [:aleth]]

(In reply to Edmund Wong (:ewong) from comment #99)
> looking back with this bug, can someone point out why we need to
> add --disable-compile-environment to mac repacks whereas the other
> platforms (that I can tell) don't have this/need this?  Is it because
> of universal build?  Even participating in this bug, and looking back,
> I don't understand this.  (It also doesn't help that I did a mess
> with the patches, so I can't follow any of the logic.)

iirc see comment 77 to 79, one can either make sure a number of compiler variables are set correctly, or make them irrelevant by disabling the compile environment. Either approach works. (In bug 1267523 I was initially hoping disable-compile-environment would also mean we wouldn't have to set TOOLTOOL_DIR, but that was mistaken. And still needs to be done for c-c/c-a repacks :-( )